{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T17:23:10Z","timestamp":1790616190328,"version":"4.1.0"},"publisher-location":"Cham","reference-count":44,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031657931","type":"print"},{"value":"9783031657948","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,1,1]],"date-time":"2024-01-01T00:00:00Z","timestamp":1704067200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,8,15]],"date-time":"2024-08-15T00:00:00Z","timestamp":1723680000000},"content-version":"vor","delay-in-days":227,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Software has emerged as a crucial tool in the current research ecosystem, frequently referenced in academic papers for its application in studies or the introduction of new software systems. Despite its prevalence, there remains a significant gap in understanding how software is cited within the scientific literature. In this study, we offer a conceptual framework for studying software citation intent and explore the use of large language models, such as BERT-based models, GPT-3.5, and GPT-4 for this task. We compile a representative software-mention dataset by merging two existing gold standard software mentions datasets and annotating them to a common citation intent scheme. This new dataset makes it possible to analyze software citation intent at the sentence level. We observe that in a fine-tuning setting, large language models can generally achieve an accuracy of over 80% on software citation intent classification on unseen, challenging data. Our research paves the way for future empirical investigations into the realm of research software, establishing a foundational framework for exploring this under-examined area.<\/jats:p>","DOI":"10.1007\/978-3-031-65794-8_6","type":"book-chapter","created":{"date-parts":[[2024,8,14]],"date-time":"2024-08-14T06:02:44Z","timestamp":1723615364000},"page":"80-99","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Scientific Software Citation Intent Classification Using Large Language Models"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7953-5168","authenticated-orcid":false,"given":"Ana-Maria","family":"Istrate","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2599-4630","authenticated-orcid":false,"given":"Joshua","family":"Fisher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8368-7773","authenticated-orcid":false,"given":"Xinyu","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2202-7006","authenticated-orcid":false,"given":"Kara","family":"Moraw","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7264-365X","authenticated-orcid":false,"given":"Kai","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3335-4537","authenticated-orcid":false,"given":"Donghui","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0130-2097","authenticated-orcid":false,"given":"Martin","family":"Klein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,8,15]]},"reference":[{"key":"6_CR1","unstructured":"Ammar, W., et\u00a0al.: Construction of the literature graph in semantic scholar. arXiv preprint arXiv:1805.02262 (2018)"},{"issue":"1","key":"6_CR2","doi-asserted-by":"publisher","first-page":"622","DOI":"10.1038\/s41597-022-01710-x","volume":"9","author":"M Barker","year":"2022","unstructured":"Barker, M., et al.: Introducing the fair principles for research software. Sci. Data 9(1), 622 (2022)","journal-title":"Sci. Data"},{"key":"6_CR3","doi-asserted-by":"publisher","unstructured":"Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615\u20133620. Association for Computational Linguistics, Hong Kong, China (2019). https:\/\/doi.org\/10.18653\/v1\/D19-1371. https:\/\/aclanthology.org\/D19-1371","DOI":"10.18653\/v1\/D19-1371"},{"key":"6_CR4","unstructured":"Bensman, S.J.: Garfield and the impact factor: the creation, utilization, and validation of a citation measure. Ann. Rev. Inf. Sci. Technol. (ARIST) 42 (2008)"},{"key":"6_CR5","unstructured":"Bird, S., et\u00a0al.: The ACL anthology reference corpus: a reference dataset for bibliographic research in computational linguistics. In: LREC (2008)"},{"issue":"3","key":"6_CR6","first-page":"325","volume":"71","author":"A Chassanoff","year":"2020","unstructured":"Chassanoff, A., Altman, M.: Curation as \u201cinteroperability with the future\u2019\u2019: preserving scholarly research software in academic libraries. J. Am. Soc. Inf. Sci. 71(3), 325\u2013337 (2020)","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"6_CR7","doi-asserted-by":"crossref","unstructured":"Cohan, A., Ammar, W., Van\u00a0Zuylen, M., Cady, F.: Structural scaffolds for citation intent classification in scientific publications. arXiv preprint arXiv:1904.01608 (2019)","DOI":"10.18653\/v1\/N19-1361"},{"issue":"1","key":"6_CR8","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1108\/eb026703","volume":"37","author":"B Cronin","year":"1981","unstructured":"Cronin, B.: The need for a theory of citing. J. Documentation 37(1), 16\u201324 (1981)","journal-title":"J. Documentation"},{"key":"6_CR9","unstructured":"Cronin, B.: The citation process. Role Significance Citations Sci. Commun. 103 (1984)"},{"issue":"6","key":"6_CR10","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1109\/MCSE.2013.133","volume":"15","author":"S Crouch","year":"2014","unstructured":"Crouch, S., et al.: The software sustainability institute: changing research software attitudes and practices. Comput. Sci. Eng. 15(6), 74\u201380 (2014)","journal-title":"Comput. Sci. Eng."},{"key":"6_CR11","unstructured":"Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2019)"},{"issue":"7","key":"6_CR12","first-page":"870","volume":"72","author":"C Du","year":"2021","unstructured":"Du, C., Cohoon, J., Lopez, P., Howison, J.: Softcite dataset: a dataset of software mentions in biomedical and economic research publications. J. Am. Soc. Inf. Sci. 72(7), 870\u2013884 (2021)","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"6_CR13","doi-asserted-by":"crossref","unstructured":"Du, C., Cohoon, J., Priem, J., Piwowar, H., Meyer, C., Howison, J.: Citeas: better software through sociotechnical change for better software citation. In: Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing, pp. 218\u2013221 (2021)","DOI":"10.1145\/3462204.3482889"},{"issue":"11","key":"6_CR14","doi-asserted-by":"publisher","first-page":"779","DOI":"10.1038\/ngeo2283","volume":"7","author":"SM Easterbrook","year":"2014","unstructured":"Easterbrook, S.M.: Open code for open science? Nat. Geosci. 7(11), 779\u2013781 (2014)","journal-title":"Nat. Geosci."},{"key":"6_CR15","unstructured":"W. Foundation: Wikimedia downloads. https:\/\/dumps.wikimedia.org"},{"issue":"3619","key":"6_CR16","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1126\/science.144.3619.649","volume":"144","author":"E Garfield","year":"1964","unstructured":"Garfield, E.: \u201cScience citation index\u2019\u2019\u2014a new dimension in indexing: this unique approach underlies versatile bibliographic systems for communicating and evaluating information. Science 144(3619), 649\u2013654 (1964)","journal-title":"Science"},{"key":"6_CR17","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1007\/BF02019306","volume":"1","author":"E Garfield","year":"1979","unstructured":"Garfield, E.: Is citation analysis a legitimate evaluation tool? Scientometrics 1, 359\u2013375 (1979)","journal-title":"Scientometrics"},{"issue":"1","key":"6_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3458754","volume":"3","author":"Y Gu","year":"2021","unstructured":"Gu, Y., et al.: Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthcare 3(1), 1\u201323 (2021)","journal-title":"ACM Trans. Comput. Healthcare"},{"key":"6_CR19","doi-asserted-by":"crossref","unstructured":"Horsfall, D., Cool, J., Hettrick, S., Pisco, A.O., Hong, N.C., Haniffa, M.: Research software engineering accelerates the translation of biomedical research for health. Nat. Med. 1\u20134 (2023)","DOI":"10.1038\/s41591-023-02353-0"},{"issue":"9","key":"6_CR20","first-page":"2137","volume":"67","author":"J Howison","year":"2016","unstructured":"Howison, J., Bullard, J.: Software in the scientific literature: problems with seeing, finding, and using software mentioned in the biology literature. J. Am. Soc. Inf. Sci. 67(9), 2137\u20132155 (2016)","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"6_CR21","unstructured":"Istrate, A.M., et al.: SoftwareImpactHackathon2023: Software Citation Intent (2023). https:\/\/github.com\/karacolada\/SoftwareImpactHackathon2023_SoftwareCitationIntent"},{"key":"6_CR22","unstructured":"Istrate, A.M., Li, D., Taraborelli, D., Torkar, M., Veytsman, B., Williams, I.: A large dataset of software mentions in the biomedical literature. arXiv preprint arXiv:2209.00693 (2022)"},{"key":"6_CR23","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1162\/tacl_a_00028","volume":"6","author":"D Jurgens","year":"2018","unstructured":"Jurgens, D., Kumar, S., Hoover, R., McFarland, D., Jurafsky, D.: Measuring the evolution of a scientific field through citation frames. Trans. Assoc. Comput. Linguist. 6, 391\u2013406 (2018)","journal-title":"Trans. Assoc. Comput. Linguist."},{"issue":"3","key":"6_CR24","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1002\/asi.5090160305","volume":"16","author":"N Kaplan","year":"1965","unstructured":"Kaplan, N.: The norms of citation behavior: prolegomena to the footnote. Am. Doc. 16(3), 179\u2013184 (1965)","journal-title":"Am. Doc."},{"key":"6_CR25","unstructured":"Latour, B.: Science in Action: How to Follow Scientists and Engineers Through Society. Harvard University Press (1987)"},{"issue":"1","key":"6_CR26","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1016\/j.joi.2019.02.007","volume":"13","author":"K Li","year":"2019","unstructured":"Li, K., Chen, P.Y., Yan, E.: Challenges of measuring software impact through citations: an examination of the lme4 R package. J. Informet. 13(1), 449\u2013461 (2019)","journal-title":"J. Informet."},{"issue":"1","key":"6_CR27","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.joi.2017.12.001","volume":"12","author":"K Li","year":"2018","unstructured":"Li, K., Yan, E.: Co-mention network of R packages: scientific impact and clustering structure. J. Informet. 12(1), 87\u2013100 (2018)","journal-title":"J. Informet."},{"issue":"4","key":"6_CR28","doi-asserted-by":"publisher","first-page":"989","DOI":"10.1016\/j.joi.2017.08.003","volume":"11","author":"K Li","year":"2017","unstructured":"Li, K., Yan, E., Feng, Y.: How is R cited in research outputs? Structure, impacts, and citation standard. J. Informet. 11(4), 989\u20131002 (2017)","journal-title":"J. Informet."},{"key":"6_CR29","unstructured":"Merton, R.K.: The Sociology of Science: Theoretical and Empirical Investigations. University of Chicago Press (1973)"},{"issue":"3","key":"6_CR30","doi-asserted-by":"publisher","first-page":"515","DOI":"10.1177\/030631288018003005","volume":"18","author":"MJ Moravcsik","year":"1988","unstructured":"Moravcsik, M.J.: Citation context classification of a citation classic concerning citation context classification. Soc. Stud. Sci. 18(3), 515\u2013521 (1988)","journal-title":"Soc. Stud. Sci."},{"issue":"3","key":"6_CR31","doi-asserted-by":"publisher","first-page":"882","DOI":"10.1162\/qss_a_00146","volume":"2","author":"JM Nicholson","year":"2021","unstructured":"Nicholson, J.M., et al.: Scite: a smart citation index that displays the context of citations and classifies their intent using deep learning. Quant. Sci. Stud. 2(3), 882\u2013898 (2021)","journal-title":"Quant. Sci. Stud."},{"key":"6_CR32","unstructured":"OpenAI: Models - OpenAI API. https:\/\/platform.openai.com\/docs\/models\/gpt-3-5-turbo"},{"key":"6_CR33","unstructured":"OpenAI: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)"},{"issue":"2","key":"6_CR34","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1016\/j.joi.2018.03.005","volume":"12","author":"X Pan","year":"2018","unstructured":"Pan, X., Yan, E., Cui, M., Hua, W.: Examining the usage, citation, and diffusion patterns of bibliometric mapping software: a comparative study of three tools. J. Informet. 12(2), 481\u2013493 (2018)","journal-title":"J. Informet."},{"issue":"11","key":"6_CR35","first-page":"1346","volume":"69","author":"H Park","year":"2018","unstructured":"Park, H., You, S., Wolfram, D.: Informal data citation for data sharing and reuse is more common than formal data citation in biomedical fields. J. Am. Soc. Inf. Sci. 69(11), 1346\u20131354 (2018)","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"6_CR36","unstructured":"Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2020)"},{"key":"6_CR37","doi-asserted-by":"crossref","unstructured":"Schindler, D., Bensmann, F., Dietze, S., Kr\u00fcger, F.: Somesci-a 5 star open data gold standard knowledge graph of software mentions in scientific articles. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 4574\u20134583 (2021)","DOI":"10.1145\/3459637.3482017"},{"key":"6_CR38","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.835","volume":"8","author":"D Schindler","year":"2022","unstructured":"Schindler, D., Bensmann, F., Dietze, S., Kr\u00fcger, F.: The role of software in science: a knowledge graph-based analysis of software mentions in pubmed central. PeerJ Comput. Sci. 8, e835 (2022)","journal-title":"PeerJ Comput. Sci."},{"key":"6_CR39","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.86","volume":"2","author":"AM Smith","year":"2016","unstructured":"Smith, A.M., Katz, D.S., Niemeyer, K.E.: Software citation principles. PeerJ Comput. Sci. 2, e86 (2016)","journal-title":"PeerJ Comput. Sci."},{"issue":"2","key":"6_CR40","doi-asserted-by":"publisher","first-page":"205395171666474","DOI":"10.1177\/2053951716664747","volume":"3","author":"J Symons","year":"2016","unstructured":"Symons, J., Alvarado, R.: Can we trust big data? applying philosophy of science to software. Big Data Soc. 3(2), 2053951716664747 (2016)","journal-title":"Big Data Soc."},{"key":"6_CR41","doi-asserted-by":"crossref","unstructured":"Teufel, S., Siddharthan, A., Tidhar, D.: Automatic classification of citation function. In: Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 103\u2013110 (2006)","DOI":"10.3115\/1610075.1610091"},{"key":"6_CR42","unstructured":"Wheeler, D.L., et\u00a0al.: Database resources of the national center for biotechnology information. Nucleic Acids Res. 36(suppl_1), D13\u2013D21 (2007)"},{"issue":"7","key":"6_CR43","doi-asserted-by":"publisher","first-page":"1490","DOI":"10.1002\/asi.22850","volume":"64","author":"G Zhang","year":"2013","unstructured":"Zhang, G., Ding, Y., Milojevi\u0107, S.: Citation content analysis (CCA): a framework for syntactic and semantic analysis of citation content. J. Am. Soc. Inform. Sci. Technol. 64(7), 1490\u20131503 (2013)","journal-title":"J. Am. Soc. Inform. Sci. Technol."},{"key":"6_CR44","doi-asserted-by":"crossref","unstructured":"Zhu, Y., et al.: Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: The IEEE International Conference on Computer Vision (ICCV) (2015)","DOI":"10.1109\/ICCV.2015.11"}],"container-title":["Lecture Notes in Computer Science","Natural Scientific Language Processing and Research Knowledge Graphs"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-65794-8_6","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,14]],"date-time":"2024-08-14T06:03:54Z","timestamp":1723615434000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-65794-8_6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"ISBN":["9783031657931","9783031657948"],"references-count":44,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-65794-8_6","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024]]},"assertion":[{"value":"15 August 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"NSLP","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Hersonissos, Crete","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Greece","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 May 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 May 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"nslp2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/nfdi4ds.github.io\/nslp2024\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}