{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,20]],"date-time":"2026-11-20T04:37:14Z","timestamp":1795149434982,"version":"4.10.3"},"reference-count":55,"publisher":"MIT Press","issue":"1","license":[{"start":{"date-parts":[[2021,2,22]],"date-time":"2021-02-22T00:00:00Z","timestamp":1613952000000},"content-version":"vor","delay-in-days":52,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100011878","name":"Vlaamse regering","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100011878","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We compare two supervised machine learning algorithms\u2014Multinomial Na\u00efve Bayes and Gradient Boosting\u2014to classify social science articles using textual data. The high level of granularity of the classification scheme used and the possibility that multiple categories are assigned to a document make this task challenging. To collect the training data, we query three discipline specific thesauri to retrieve articles corresponding to specialties in the classification. The resulting data set consists of 113,909 records and covers 245 specialties, aggregated into 31 subdisciplines from three disciplines. Experts were consulted to validate the thesauri-based classification. The resulting multilabel data set is used to train the machine learning algorithms in different configurations. We deploy a multilabel classifier chaining model, allowing for an arbitrary number of categories to be assigned to each document. The best results are obtained with Gradient Boosting. The approach does not rely on citation data. It can be applied in settings where such information is not available. We conclude that fine-grained text-based classification of social sciences publications at a subdisciplinary level is a hard task, for humans and machines alike. A combination of human expertise and machine learning is suggested as a way forward to improve the classification of social sciences documents.<\/jats:p>","DOI":"10.1162\/qss_a_00106","type":"journal-article","created":{"date-parts":[[2020,12,30]],"date-time":"2020-12-30T10:26:22Z","timestamp":1609323982000},"page":"89-110","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":26,"title":["Fine-grained classification of social science journal articles using\n                    textual data: A comparison of supervised machine learning\n                    approaches"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1680-0112","authenticated-orcid":true,"given":"Joshua","family":"Eykens","sequence":"first","affiliation":[{"name":"Centre for R&D Monitoring (ECOOM), Faculty of Social Sciences, University of Antwerp, Middelheimlaan 1, 2020 Antwerp, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3129-0330","authenticated-orcid":true,"given":"Raf","family":"Guns","sequence":"additional","affiliation":[{"name":"Centre for R&D Monitoring (ECOOM), Faculty of Social Sciences, University of Antwerp, Middelheimlaan 1, 2020 Antwerp, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4869-7949","authenticated-orcid":true,"given":"Tim C. E.","family":"Engels","sequence":"additional","affiliation":[{"name":"Centre for R&D Monitoring (ECOOM), Faculty of Social Sciences, University of Antwerp, Middelheimlaan 1, 2020 Antwerp, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,4,8]]},"reference":[{"issue":"3","key":"2021041320043709900_bib55","doi-asserted-by":"crossref","first-page":"e18029","DOI":"10.1371\/journal.pone.0018029","article-title":"Clustering more than two million\n                        biomedical publications: Comparing the accuracies of nine text-based\n                        similarity approaches","volume":"6","author":"Boyack","year":"2011","journal-title":"PLoS ONE"},{"issue":"3","key":"2021041320043709900_bib1","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1016\/j.joi.2014.04.001","article-title":"Including cited non-source items in a\n                        large-scale map of science: What difference does it make?","volume":"8","author":"Boyack","year":"2014","journal-title":"Journal of Informetrics"},{"issue":"1","key":"2021041320043709900_bib2","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1007\/BF02019280","article-title":"Co-word analysis as a tool for\n                        describing the network of interactions between basic and technological\n                        research: The case of polymer chemistry","volume":"22","author":"Callon","year":"1991","journal-title":"Scientometrics"},{"issue":"2","key":"2021041320043709900_bib3","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1177\/053901883022002003","article-title":"From translations to problematic\n                        networks: An introduction to co-word analysis","volume":"22","author":"Callon","year":"1983","journal-title":"Social Science Information"},{"key":"2021041320043709900_bib4","article-title":"Identifying the development and\n                        application of artificial intelligence in scientific text","author":"Dunham","year":"2020","journal-title":"arXiv preprint"},{"key":"2021041320043709900_bib5","article-title":"Supervised classification of SSH\n                        publications","author":"Eykens","year":"2020"},{"key":"2021041320043709900_bib6","first-page":"738","article-title":"Article level classification of publications in sociology: An\n                        experimental assessment of supervised machine learning\n                        approaches","volume-title":"17th International Conference on Scientometrics &\n                        Informetrics (ISSI2019)","author":"Eykens","year":"2019"},{"issue":"2","key":"2021041320043709900_bib7","first-page":"176","article-title":"Indexing consistency in MEDLINE","volume":"71","author":"Funk","year":"1983","journal-title":"Bulletin of the Medical Library Association"},{"issue":"3","key":"2021041320043709900_bib8","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1007\/BF02458488","article-title":"An item-by-item subject classification\n                        of papers published in multidisciplinary and general journals using\n                        reference analysis","volume":"44","author":"Gl\u00e4nzel","year":"1999","journal-title":"Scientometrics"},{"issue":"2","key":"2021041320043709900_bib9","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1007\/s11192-018-2775-x","article-title":"A comparison of cognitive and organizational classification\n                        of publications in the social sciences and humanities","volume":"116","author":"Guns","year":"2018","journal-title":"Scientometrics"},{"key":"2021041320043709900_bib10","first-page":"197","article-title":"What is a discipline? The\n                        conceptualization of research areas and their operationalization in\n                        bibliometric research","volume-title":"Science, Technology and Innovation Indicators in\n                        Transition\u2014STI2018","author":"Hammarfelt","year":"2018"},{"key":"2021041320043709900_bib11","article-title":"spaCy 2.0.11","author":"Honnibal","year":"2018"},{"issue":"6","key":"2021041320043709900_bib12","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1016\/j.ipm.2009.06.003","article-title":"Hybrid clustering for validation and\n                        improvement of subject-classification schemes","volume":"45","author":"Janssens","year":"2009","journal-title":"Information Processing and Management"},{"key":"2021041320043709900_bib13","article-title":"Large scale subject category\n                        classification of scholarly papers with deep attentive neural\n                        networks","author":"Kandimalla","year":"2020","journal-title":"arXiv preprint"},{"key":"2021041320043709900_bib14","first-page":"1","article-title":"LightGBM: A highly efficient gradient\n                        boosting decision tree","volume-title":"Neural Information Processing Systems 2017","author":"Ke","year":"2017"},{"key":"2021041320043709900_bib15","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1007\/s11192-018-2711-0","article-title":"Publication patterns in the social\n                        sciences and humanities: Evidence from eight European\n                        countries","volume":"116","author":"Kulczycki","year":"2018","journal-title":"Scientometrics"},{"issue":"4","key":"2021041320043709900_bib16","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1002\/jrsm.1317","article-title":"Discriminating between empirical studies\n                        and nonempirical works using automated text classification","volume":"9","author":"Langlois","year":"2018","journal-title":"Research Synthesis Methods"},{"issue":"1","key":"2021041320043709900_bib17","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1177\/096100060003200102","article-title":"Interindexer consistency in\n                        PsycINFO","volume":"32","author":"Leininger","year":"2000","journal-title":"Journal of Librarianship and Information\n                        Science"},{"key":"2021041320043709900_bib18","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1007\/BFb0026666","article-title":"Na\u00efve (Bayes) at forty: The independence assumption in\n                        information retrieval","volume-title":"10th European Conference on Machine Learning\u2014ECML-98","author":"Lewis","year":"1998"},{"key":"2021041320043709900_bib19","first-page":"63","article-title":"NLTK: The Natural Language\n                        Toolkit","volume-title":"Proceedings of the ACL-02 Workshop on\n                        Effective tools and methodologies for teaching natural language processing\n                        and computational linguistics","author":"Loper","year":"2002"},{"issue":"5","key":"2021041320043709900_bib20","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1136\/amiajnl-2012-001072","article-title":"Direct comparison between support vector\n                        machine and multinomial naive Bayes algorithms for medical abstract\n                        classification","volume":"19","author":"Matwin","year":"2012","journal-title":"Journal of the American Medical\n                        Informatics Association"},{"key":"2021041320043709900_bib21","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1145\/1141753.1141819","article-title":"Measuring inter-indexer consistency using a\n                        thesaurus","volume-title":"6th ACM\/IEEE-CS joint conference on\n                        Digital libraries","author":"Medelyan","year":"2006"},{"key":"2021041320043709900_bib22","first-page":"181","article-title":"Complex linguistic features for text\n                        classification: A comprehensive study","volume-title":"Advances in Information Retrieval. ECIR 2004","author":"Moschitti","year":"2004"},{"issue":"1","key":"2021041320043709900_bib23","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/s11192-006-0007-2","article-title":"Bibliometric monitoring of research performance in the social\n                        sciences and humanities: A review","volume":"66","author":"Nederhof","year":"2006","journal-title":"Scientometrics"},{"issue":"2","key":"2021041320043709900_bib24","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.joi.2006.12.001","article-title":"Comparing alternatives to the Web of\n                        Science for coverage of the social sciences\u2019\n                        literature","volume":"1","author":"Norris","year":"2007","journal-title":"Journal of Informetrics"},{"key":"2021041320043709900_bib25","volume-title":"Revised Fields of Science and Technology (FOS)\n                        Classification in the Frascati Manual","author":"OECD","year":"2007"},{"issue":"4","key":"2021041320043709900_bib26","doi-asserted-by":"crossref","first-page":"280","DOI":"10.1093\/reseval\/rvs019","article-title":"The representation of the social\n                        sciences and humanities in the Web of Science\u2014A comparison of\n                        publication patterns and incentive structures in Flanders and Norway\n                        (2005\u20139)","volume":"21","author":"Ossenblok","year":"2012","journal-title":"Research Evaluation"},{"issue":"85","key":"2021041320043709900_bib27","first-page":"2825","article-title":"Scikit-learn: Machine Learning in\n                        Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning\n                        Research"},{"key":"2021041320043709900_bib28","unstructured":"Read,\n                            J.\n          \n                        (2010). Scalable Multi-label Classification.\n                        Doctoral thesis, University of\n                        Waikato, Hamilton, New\n                        Zealand. Retrieved from https:\/\/hdl.handle.net\/10289\/4645"},{"key":"2021041320043709900_bib29","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-04174-7_17","article-title":"Classifier chains for multi-label\n                        classification","volume-title":"Machine Learning and Knowledge Discovery in Databases. ECML PKDD\n                        2009","author":"Read","year":"2009"},{"key":"2021041320043709900_bib30","first-page":"616","article-title":"Tackling the poor assumptions of na\u00efve Bayes text\n                        classifiers","volume-title":"Twentieth International Conference on Machine Learning\n                        (ICML-2003)","author":"Rennie","year":"2003"},{"issue":"6","key":"2021041320043709900_bib31","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1007\/BF02025827","article-title":"Co-word maps of biotechnology: An\n                        example of cognitive scientometrics","volume":"6","author":"Rip","year":"1984","journal-title":"Scientometrics"},{"issue":"2","key":"2021041320043709900_bib32","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1016\/0306-4573(81)90028-5","article-title":"Indexing consistency, quality and\n                        efficiency","volume":"17","author":"Rollin","year":"1981","journal-title":"Information Processing &\n                        Management"},{"issue":"2\/3","key":"2021041320043709900_bib56","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1023\/A:1007649029923","article-title":"Boost-exter: A boosting-based system for\n                        text categorization","volume":"39","author":"Schapire","year":"2000","journal-title":"Machine Learning"},{"issue":"1","key":"2021041320043709900_bib33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/505282.505283","article-title":"Machine learning in automated text\n                        categorization","volume":"34","author":"Sebastiani","year":"2002","journal-title":"ACM Computer Surveys"},{"key":"2021041320043709900_bib34","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-23808-6_10","article-title":"On the stratification of multi-label\n                        data","volume-title":"Machine Learning and Knowledge Discovery in Databases. Joint\n                        European Conference on Machine Learning and Knowledge Discovery in\n                        Databases\u2014ECML PKDD 2011","author":"Sechidis","year":"2011"},{"issue":"5","key":"2021041320043709900_bib35","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1016\/j.tibs.2009.02.002","article-title":"Four stages of a scientific discipline; four types of\n                        scientist","volume":"34","author":"Shneider","year":"2009","journal-title":"Trends in Biochemical Sciences"},{"issue":"1","key":"2021041320043709900_bib36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/(SICI)1097-4571(199101)42:1<1::AID-ASI1>3.0.CO;2-9","article-title":"Indexing consistency in Information Science\n                        Abstracts","volume":"42","author":"Sievert","year":"1991","journal-title":"Journal of the American Society for\n                        Information Science"},{"issue":"1","key":"2021041320043709900_bib37","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1016\/j.joi.2017.12.006","article-title":"Granularity of algorithmically\n                        constructed publication-level classifications of research publications:\n                        Identification of topics","volume":"12","author":"Sj\u00f6g\u00e5rde","year":"2018","journal-title":"Journal of\n                        Informetrics"},{"issue":"1","key":"2021041320043709900_bib38","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1162\/qss_a_00004","article-title":"Granularity of algorithmically\n                        constructed publication-level classifications of research publications:\n                        Identification of specialties","volume":"1","author":"Sj\u00f6g\u00e5rde","year":"2019","journal-title":"Quantitative Studies\n                        of Science"},{"key":"2021041320043709900_bib57","unstructured":"Sorower, M.\n                                S.\n           (2010).\n                        A literature survey on algorithms for multi-label\n                        learning. Technical Report,\n                        Corvallis: Oregon State\n                        University."},{"issue":"1","key":"2021041320043709900_bib39","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1017\/S0269889700001071","article-title":"The sociology of scientific disciplines:\n                        On the genesis and stability of the disciplinary structure of modern\n                        science","volume":"5","author":"Stichweh","year":"1992","journal-title":"Science in Context"},{"key":"2021041320043709900_bib40","first-page":"82","article-title":"Differentiation in science: Causes and\n                        consequences","volume-title":"Unity of Knowledge in Transdisciplinary Research for Sustainable\n                        Development","author":"Stichweh","year":"2003"},{"issue":"4","key":"2021041320043709900_bib41","doi-asserted-by":"crossref","first-page":"775","DOI":"10.1108\/JD-06-2014-0082","article-title":"The kaleidoscope of\n                        disciplinarity","volume":"77","author":"Sugimoto","year":"2015","journal-title":"Journal of Documentation"},{"issue":"10","key":"2021041320043709900_bib42","doi-asserted-by":"crossref","first-page":"2464","DOI":"10.1002\/asi.23596","article-title":"Map of science with topic modeling:\n                        Comparison of unsupervised learning and human-assigned subject\n                        classification","volume":"67","author":"Suominen","year":"2016","journal-title":"Journal of the Association for\n                        Information Science and Technology"},{"issue":"1","key":"2021041320043709900_bib43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18352\/lq.10285","article-title":"Annif: DIY automated subject indexing\n                        using multiple algorithms","volume":"29","author":"Suominen","year":"2019","journal-title":"LIBER Quarterly"},{"issue":"3","key":"2021041320043709900_bib58","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4018\/jdwm.2007070101","article-title":"Multi-label classification: An\n                        overview","volume":"3","author":"Tsoumakas","year":"2007","journal-title":"International Journal for Data Warehousing\n                        and Mining"},{"issue":"3","key":"2021041320043709900_bib44","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1007\/s11192-006-0118-9","article-title":"Mapping research topics using\n                        word-reference co-occurrences: A method and an exploratory case\n                        study","volume":"68","author":"van den Besselaar","year":"2006","journal-title":"Scientometrics"},{"key":"2021041320043709900_bib45","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1016\/j.procs.2019.01.075","article-title":"The creation of the Flemish research\n                        discipline list, an important step forward in harmonising research\n                        information (systems)","volume":"146","author":"Vancauwenbergh","year":"2019","journal-title":"Procedia Computer\n                        Science"},{"key":"2021041320043709900_bib46","doi-asserted-by":"crossref","first-page":"354","DOI":"10.5771\/0943-7444-2019-5-354","article-title":"The Flemish research discipline\n                        classification standard: A practical approach","volume":"46","author":"Vancauwenbergh","year":"2019","journal-title":"Knowledge Organisation"},{"issue":"12","key":"2021041320043709900_bib47","doi-asserted-by":"crossref","first-page":"2378","DOI":"10.1002\/asi.22748","article-title":"A New Methodology for Constructing a Publication-Level\n                        Classification System of Science","volume":"63","author":"Waltman","year":"2012","journal-title":"Journal of the\n                        American Society for Information Science and Technology"},{"issue":"1","key":"2021041320043709900_bib48","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.joi.2011.10.001","article-title":"Topics in dynamic research communities: An exploratory study\n                        for the field of information retrieval","volume":"6","author":"Yan","year":"2012","journal-title":"Journal of\n                        Informetrics"},{"key":"2021041320043709900_bib49","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1007\/s11192-014-1321-8","article-title":"Clustering scientific documents with\n                        topic modeling","volume":"100","author":"Yau","year":"2014","journal-title":"Scientometrics"},{"key":"2021041320043709900_bib50","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/s11704-017-7031-7","article-title":"Binary relevance for multi-label\n                        learning: an overview","volume":"12","author":"Zhang","year":"2018","journal-title":"Frontiers of Computer\n                        Science"},{"issue":"8","key":"2021041320043709900_bib51","doi-asserted-by":"crossref","first-page":"1819","DOI":"10.1109\/TKDE.2013.39","article-title":"A review on multi-label learning\n                        algorithms","volume":"26","author":"Zhang","year":"2014","journal-title":"IEEE Transactions on Knowledge and Data\n                        Engineering"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/qss\/article-pdf\/2\/1\/89\/1906557\/qss_a_00106.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/qss\/article-pdf\/2\/1\/89\/1906557\/qss_a_00106.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,14]],"date-time":"2021-04-14T20:07:44Z","timestamp":1618430864000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/1\/89\/97077\/Fine-grained-classification-of-social-science"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":55,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,4,8]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00106","relation":{},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}