{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T07:44:57Z","timestamp":1780386297890,"version":"3.54.1"},"reference-count":51,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T00:00:00Z","timestamp":1668470400000},"content-version":"vor","delay-in-days":318,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,12,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Subject area classification is an important first phase in the entire process involved in bibliometrics. In this paper, we explore the possibility of using automated algorithms for classifying scientific papers related to Artificial Intelligence at the document level. The current process is semimanual and journal based, a realization that, we argue, opens up the potential for inaccuracies. To counter this, our proposed automated approach makes use of neural networks, specifically BERT. The classification accuracy of our model reaches 96.5%. In addition, the model was used for further classifying documents from 26 different subject areas from the Scopus database. Our findings indicate that a significant subset of existing Computer Science, Decision Science, and Mathematics publications could potentially be classified as AI-related. The same holds in particular cases in other science fields such as Medicine and Psychology or Arts and Humanities. The above indicate that in subject area classification processes, there is room for automatic approaches to be utilized in a complementary manner with traditional manual procedures.<\/jats:p>","DOI":"10.1162\/qss_a_00223","type":"journal-article","created":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T15:10:19Z","timestamp":1668525019000},"page":"1119-1132","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":13,"title":["AI for AI: Using AI methods for classifying AI science documents"],"prefix":"10.1162","volume":"3","author":[{"given":"Evi","family":"Sachini","sequence":"first","affiliation":[{"name":"National Documentation Centre (EKT), Palaio Faliro, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2882-8307","authenticated-orcid":true,"given":"Konstantinos","family":"Sioumalas-Christodoulou","sequence":"additional","affiliation":[{"name":"National Documentation Centre (EKT), Palaio Faliro, Greece"},{"name":"Department of History and Philosophy of Science, National and Kapodistrian University of Athens, Athens, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefanos","family":"Christopoulos","sequence":"additional","affiliation":[{"name":"National Documentation Centre (EKT), Palaio Faliro, Greece"},{"name":"Cadence Design Systems, 85622 Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikolaos","family":"Karampekios","sequence":"additional","affiliation":[{"name":"National Documentation Centre (EKT), Palaio Faliro, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,12,20]]},"reference":[{"key":"2023020920354506500_bib1","first-page":"66","article-title":"Towards a multilingual, comprehensive and open scientific journal ontology","volume-title":"Proceedings of the 13th International Conference of the International Society for Scientometrics and Informetrics (ISSI)","author":"Archambault","year":"2011"},{"key":"2023020920354506500_bib2","author":"arXiv","year":"2022"},{"key":"2023020920354506500_bib3","volume-title":"Moderators","author":"arXiv","year":"2022"},{"issue":"1","key":"2023020920354506500_bib4","article-title":"An overview of graph-based keyword extraction methods and approaches","volume":"39","author":"Beliga","year":"2015","journal-title":"Journal of Information and Organizational Sciences"},{"key":"2023020920354506500_bib6","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4980-1","volume-title":"On knowledge base management systems: Integrating artificial intelligence and database technologies","author":"Brodie","year":"2012"},{"key":"2023020920354506500_bib7","unstructured":"Bryant, R. E.\n           (1997). Discovery and decision: Exploring the metaphysics and epistemology of scientific classification. Doctoral dissertation, University of Edinburgh."},{"issue":"6","key":"2023020920354506500_bib8","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1002\/asi.4630240604","article-title":"Clustering of scientific journals","volume":"24","author":"Carpenter","year":"1973","journal-title":"Journal of the American Society for Information Science"},{"key":"2023020920354506500_bib9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/IETC47856.2020.9249211","article-title":"Research paper classification using supervised machine learning techniques","volume-title":"2020 Intermountain Engineering, Technology and Computing (IETC)","author":"Chowdhury","year":"2020"},{"key":"2023020920354506500_bib10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2019","journal-title":"arXiv:1810.04805"},{"key":"2023020920354506500_bib11","article-title":"Classification of the sciences: The nineteenth century tradition","volume-title":"Classifications in their social context","author":"Dolby","year":"1979"},{"key":"2023020920354506500_bib12","volume-title":"Primitive classification","author":"Durkheim","year":"1963"},{"issue":"3","key":"2023020920354506500_bib13","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1023\/A:1022378804087","article-title":"A new classification scheme of science fields and subfields designed for scientometric evaluation purposes","volume":"56","author":"Gl\u00e4nzel","year":"2003","journal-title":"Scientometrics"},{"issue":"2","key":"2023020920354506500_bib14","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1007\/BF02018480","article-title":"Coping with the problem of subject classification diversity","volume":"35","author":"G\u00f3mez","year":"1996","journal-title":"Scientometrics"},{"key":"2023020920354506500_bib15","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1007\/s11192-011-0485-8","article-title":"Improving SCImago Journal & Country Rank (SJR) subject classification through reference analysis","volume":"89","author":"G\u00f3mez-N\u00fa\u00f1ez","year":"2011","journal-title":"Scientometrics"},{"key":"2023020920354506500_bib16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.13012","article-title":"Comparing BERT against traditional machine learning text classification","author":"Gonz\u00e1lez-Carvajal","year":"2020","journal-title":"arXiv:2005.13012"},{"issue":"4","key":"2023020920354506500_bib17","doi-asserted-by":"publisher","first-page":"984","DOI":"10.1002\/asi.23734","article-title":"Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge?","volume":"68","author":"Klavans","year":"2017","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"2023020920354506500_bib18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.11942","article-title":"ALBERT: A lite BERT for self-supervised learning of language representations","author":"Lan","year":"2020","journal-title":"arXiv:1909.11942"},{"issue":"2","key":"2023020920354506500_bib19","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1002\/asi.20967","article-title":"A global map of science based on the ISI subject categories","volume":"60","author":"Leydesdorff","year":"2009","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"2023020920354506500_bib21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv:1907.11692"},{"issue":"1","key":"2023020920354506500_bib23","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1162\/qss_a_00014","article-title":"Practical method to reclassify Web of Science articles into unique subject categories and broad disciplines","volume":"1","author":"Milojevi\u0107","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"3","key":"2023020920354506500_bib24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3439726","article-title":"Deep learning-based text classification: A comprehensive review","volume":"54","author":"Minaee","year":"2021","journal-title":"ACM Computing Surveys"},{"key":"2023020920354506500_bib25","volume-title":"Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity","author":"Narin","year":"1976"},{"issue":"1","key":"2023020920354506500_bib26","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1002\/asi.4630270104","article-title":"Structure of the biomedical literature","volume":"27","author":"Narin","year":"1976","journal-title":"Journal of the American Society for Information Science"},{"key":"2023020920354506500_bib27","volume-title":"OECD category scheme","author":"OECD","year":"2021"},{"issue":"sup1","key":"2023020920354506500_bib28","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1080\/00031305.2018.1518265","article-title":"Expert knowledge elicitation: Subjective but scientific","volume":"73","author":"O\u2019Hagan","year":"2019","journal-title":"The American Statistician"},{"issue":"1","key":"2023020920354506500_bib29","doi-asserted-by":"publisher","first-page":"122","DOI":"10.1162\/qss_a_00174","article-title":"Scientometric engineering: Exploring citation dynamics via arXiv eprints","volume":"3","author":"Okamura","year":"2022","journal-title":"Quantitative Science Studies"},{"key":"2023020920354506500_bib30","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1016\/j.eswa.2016.03.045","article-title":"Ensemble of keyword extraction methods and classifiers in text classification","volume":"57","author":"Onan","year":"2016","journal-title":"Expert Systems with Applications"},{"issue":"4","key":"2023020920354506500_bib31","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1177\/003754978704800403","article-title":"Artificial intelligence in modelling and simulation: Directions to explore","volume":"48","author":"\u00d6ren","year":"1987","journal-title":"Simulation"},{"issue":"13","key":"2023020920354506500_bib32","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1002\/asi.10153","article-title":"Algorithmic procedure for finding semantically related journals","volume":"53","author":"Pudovkin","year":"2002","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"2023020920354506500_bib33","author":"Radford","year":"2018","journal-title":"Improving language understanding by generative pre-training"},{"issue":"9","key":"2023020920354506500_bib34","doi-asserted-by":"publisher","first-page":"1823","DOI":"10.1002\/asi.21086","article-title":"Content-based and algorithmic classifications of journals: Perspectives on the dynamics of scientific communication and indexer effects","volume":"60","author":"Rafols","year":"2009","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"2023020920354506500_bib35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1705.10694","article-title":"Deep learning is robust to massive label noise","author":"Rolnick","year":"2017","journal-title":"arXiv:1705.10694"},{"key":"2023020920354506500_bib36","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.7223811","article-title":"Model and code of the scientific paper \u201cAI for AI: Using AI methods for classifying AI science documents.\u201d","author":"Sachini","year":"2022","journal-title":"Zenodo"},{"key":"2023020920354506500_bib37","volume-title":"Content coverage guide","author":"Scopus","year":"2020"},{"key":"2023020920354506500_bib38","volume-title":"What is the complete list of Scopus Subject Areas and All Science Journal Classification Codes (ASJC)?","author":"Scopus","year":"2021"},{"key":"2023020920354506500_bib39","volume-title":"Content policy and selection","author":"Scopus","year":"2021"},{"key":"2023020920354506500_bib40","volume-title":"What are the most used subject area categories and classifications in Scopus?","author":"Scopus","year":"2021"},{"issue":"1","key":"2023020920354506500_bib41","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1214\/ss\/1177013426","article-title":"Probability judgment in artificial intelligence and expert systems","volume":"2","author":"Shafer","year":"1987","journal-title":"Statistical Science"},{"key":"2023020920354506500_bib42","doi-asserted-by":"publisher","DOI":"10.1201\/9781420051773","volume-title":"Intelligent systems: Modeling, optimization, and control","author":"Shin","year":"2017"},{"issue":"1","key":"2023020920354506500_bib43","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1016\/j.joi.2018.12.005","article-title":"Comparing journal and paper level classifications of science","volume":"13","author":"Shu","year":"2019","journal-title":"Journal of Informetrics"},{"issue":"6","key":"2023020920354506500_bib44","doi-asserted-by":"publisher","first-page":"e0233977","DOI":"10.1371\/journal.pone.0233977","article-title":"Subjective data, objective data and the role of bias in predictive modelling: Lessons from a dispositional learning analytics application","volume":"15","author":"Tempelaar","year":"2020","journal-title":"PLOS ONE"},{"key":"2023020920354506500_bib45","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.03762","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"arXiv:1706.03762"},{"key":"2023020920354506500_bib46","volume-title":"Classification and indexing in science","author":"Vickery","year":"1958"},{"issue":"2","key":"2023020920354506500_bib47","doi-asserted-by":"publisher","first-page":"662","DOI":"10.1162\/qss_a_00125","article-title":"Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome","volume":"2","author":"Vincent-Lamarre","year":"2021","journal-title":"Quantitative Science Studies"},{"issue":"12","key":"2023020920354506500_bib54","doi-asserted-by":"publisher","first-page":"2378","DOI":"10.1002\/asi.22748","article-title":"A new methodology for constructing a publication-level classification system of science","volume":"63","author":"Waltman","year":"2012","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"3","key":"2023020920354506500_bib48","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1002\/leap.1229","article-title":"A conceptual peer review model for arXiv and other preprint databases","volume":"32","author":"Wang","year":"2019","journal-title":"Learned Publishing"},{"issue":"2","key":"2023020920354506500_bib49","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1016\/j.joi.2016.02.003","article-title":"Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus","volume":"10","author":"Wang","year":"2016","journal-title":"Journal of Informetrics"},{"key":"2023020920354506500_bib50","volume-title":"Research area schemas","author":"Web of Science","year":"2020"},{"issue":"5","key":"2023020920354506500_bib51","doi-asserted-by":"publisher","first-page":"1217","DOI":"10.1002\/acp.3677","article-title":"Evidence of vulnerability to decision bias in expert field scientists","volume":"34","author":"Wilson","year":"2020","journal-title":"Applied Cognitive Psychology"},{"key":"2023020920354506500_bib52","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1609.08144","article-title":"Google\u2019s neural machine translation system: Bridging the gap between human and machine translation","author":"Wu","year":"2016","journal-title":"arXiv:1609.08144"},{"key":"2023020920354506500_bib53","doi-asserted-by":"publisher","DOI":"10.1787\/7b43b038-en","article-title":"Measuring the AI content of government-funded R&D projects: A proof of concept for the OECD Fundstat initiative","volume-title":"OECD Science, Technology and Industry Working Papers","author":"Yamashita","year":"2021"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/4\/1119\/2070737\/qss_a_00223.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/4\/1119\/2070737\/qss_a_00223.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T15:37:10Z","timestamp":1675957030000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/3\/4\/1119\/113767\/AI-for-AI-Using-AI-methods-for-classifying-AI"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":51,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12,20]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00223","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v3\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00223\/v3\/review1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}