{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,9]],"date-time":"2026-06-09T21:54:21Z","timestamp":1781042061617,"version":"3.54.1"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,7,18]],"date-time":"2020-07-18T00:00:00Z","timestamp":1595030400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,18]],"date-time":"2020-07-18T00:00:00Z","timestamp":1595030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001381","name":"National Research Foundation Singapore","doi-asserted-by":"publisher","award":["NRF2014-NRF-SRIE001-027"],"award-info":[{"award-number":["NRF2014-NRF-SRIE001-027"]}],"id":[{"id":"10.13039\/501100001381","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We study whether humans or machine learning (ML) classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. We recruit both undergraduate and postgraduate assistants for this task in separate stages, and compare their performance against the support vectors machine ML algorithm at classifying European Research Council Starting Grant project abstracts to their actual evaluation panels, which are organised by discipline groups. On average, ML is more accurate than human classifiers, across a variety of training and test datasets, and across evaluation panels. ML classifiers trained on different training sets are also more reliable than human classifiers, meaning that different ML classifiers are more consistent in assigning the same classifications to any given abstract, compared to different human classifiers. While the top five percentile of human classifiers can outperform ML in limited cases, selection and training of such classifiers is likely costly and difficult compared to training ML models. Our results suggest ML models are a cost effective and highly accurate method for addressing problems in comparative bibliometric analysis, such as harmonising the discipline classifications of research from different funding agencies or countries.<\/jats:p>","DOI":"10.1007\/s11192-020-03614-2","type":"journal-article","created":{"date-parts":[[2020,7,18]],"date-time":"2020-07-18T02:02:24Z","timestamp":1595037744000},"page":"1197-1212","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":44,"title":["Evaluating human versus machine learning performance in classifying research abstracts"],"prefix":"10.1007","volume":"125","author":[{"given":"Yeow Chong","family":"Goh","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xin Qing","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Walter","family":"Theseira","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giovanni","family":"Ko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Khiam Aik","family":"Khor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,7,18]]},"reference":[{"key":"3614_CR2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-3223-4_6","volume-title":"Mining text data","author":"CC Aggarwal","year":"2012","unstructured":"Aggarwal, C. C., & Zhai, C. (2012). A survey of text classification algorithms. In C. Aggarwal & C. Zhai (Eds.), Mining text data. Boston, MA: Springer. https:\/\/doi.org\/10.1007\/978-1-4614-3223-4_6."},{"key":"3614_CR3","volume-title":"Modern information retrieval","author":"R Baeza-Yates","year":"1999","unstructured":"Baeza-Yates, R., & Ribeiro-Neto, B. (1999). Modern information retrieval. New York, NY: ACM Press."},{"key":"3614_CR4","first-page":"281","volume":"13","author":"J Bergstra","year":"2012","unstructured":"Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13, 281\u2013305.","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"3614_CR5","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1002\/(sici)1097-4571(199105)42:4<233::aid-asi1>3.0.co;2-i","volume":"42","author":"RR Braam","year":"1991","unstructured":"Braam, R. R., Moed, H. F., & Van Raan, A. F. (1991a). Mapping of science by combined co-citation and word analysis. I. Structural aspects. Journal of the American Society for information science, 42(4), 233\u2013251.","journal-title":"Journal of the American Society for information science"},{"issue":"4","key":"3614_CR6","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1002\/(SICI)1097-4571(199105)42:4<252::AID-ASI2>3.0.CO;2-G","volume":"42","author":"RR Braam","year":"1991","unstructured":"Braam, R. R., Moed, H. F., & Van Raan, A. F. (1991b). Mapping of science by combined co-citation and word analysis. II: Dynamical aspects. Journal of the American Society for information science, 42(4), 252\u2013266.","journal-title":"Journal of the American Society for information science"},{"issue":"2","key":"3614_CR7","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1177\/053901883022002003","volume":"22","author":"M Callon","year":"1983","unstructured":"Callon, M., Courtial, J. P., Turner, W. A., & Bauin, S. (1983). From translations to problematic networks: An introduction to co-word analysis. Information (International Social Science Council), 22(2), 191\u2013235. https:\/\/doi.org\/10.1177\/053901883022002003.","journal-title":"Information (International Social Science Council)"},{"issue":"3","key":"3614_CR8","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/BF00994018","volume":"20","author":"C Cortes","year":"1995","unstructured":"Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine learning, 20(3), 273\u2013297. https:\/\/doi.org\/10.1007\/BF00994018.","journal-title":"Machine learning"},{"issue":"1","key":"3614_CR9","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/BF02016793","volume":"26","author":"R De Bruin","year":"1993","unstructured":"De Bruin, R., & Moed, H. (1993). Delimitation of scientific subfields using cognitive words from corporate addresses in scientific publications. Scientometrics, 26(1), 65\u201380. https:\/\/doi.org\/10.1007\/BF02016793.","journal-title":"Scientometrics"},{"key":"3614_CR10","unstructured":"European Research Council. (2019a). 2019 ERC evaluation panels and keywords. Retrieved from https:\/\/erc.europa.eu\/content\/erc-panel-structure-2019."},{"key":"3614_CR11","unstructured":"European Research Council. (2019b). Information for applicants to the starting and consolidator grant 2020 calls. Retrieved from https:\/\/ec.europa.eu\/research\/participants\/data\/ref\/h2020\/other\/guides_for_applicants\/h2020-guide20-erc-stg-cog_en.pdf."},{"issue":"5","key":"3614_CR12","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378. https:\/\/doi.org\/10.1037\/h0031619.","journal-title":"Psychological Bulletin"},{"issue":"4","key":"3614_CR13","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1093\/reseval\/rvw016","volume":"25","author":"CA Freyman","year":"2016","unstructured":"Freyman, C. A., Byrnes, J. J., & Alexander, J. (2016). Machine-learning-based classification of research grant award records. Research Evaluation, 25(4), 442\u2013450. https:\/\/doi.org\/10.1093\/reseval\/rvw016.","journal-title":"Research Evaluation"},{"key":"3614_CR14","unstructured":"Herzog, C., Sorensen, A., & Taylor, M. (2016). Forward-looking analysis based on grants data and machine learning based research classifications as an analytical tool. Paper presented at OECD Blue Sky 2016 Forum, Ghent, Belgium. Retrieved from https:\/\/www.oecd.org\/sti\/093%20-%20OECDForward-lookinganalysisbasedongrantsdataandmachinelearningbasedresearchclassificationsasananalyticaltool%20(1).pdf."},{"key":"3614_CR15","unstructured":"Khor, K. A., Ko, G., & Walter, T. (2018). Applying machine learning to compare research grant programs. In 23rd International conference on science and technology indicators (STI 2018) conference proceedings, September 12\u201314, 2018. Leiden: Centre for Science and Technology Studies (CWTS)."},{"issue":"5","key":"3614_CR17","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1177\/016555158701300501","volume":"13","author":"J King","year":"1987","unstructured":"King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of Information Science, 13(5), 261\u2013276. https:\/\/doi.org\/10.1177\/016555158701300501.","journal-title":"Journal of Information Science"},{"issue":"6997","key":"3614_CR18","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1038\/430311a","volume":"430","author":"DA King","year":"2004","unstructured":"King, D. A. (2004). The scientific impact of nations. Nature, 430(6997), 311. https:\/\/doi.org\/10.1038\/430311a.","journal-title":"Nature"},{"key":"3614_CR20","doi-asserted-by":"publisher","DOI":"10.2307\/2529786","author":"JR Landis","year":"1977","unstructured":"Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. https:\/\/doi.org\/10.2307\/2529786.","journal-title":"Biometrics"},{"issue":"2","key":"3614_CR21","doi-asserted-by":"publisher","first-page":"2400","DOI":"10.1016\/j.eswa.2007.12.052","volume":"36","author":"CH Lee","year":"2009","unstructured":"Lee, C. H., & Yang, H. C. (2009). Construction of supervised and unsupervised learning systems for multilingual text categorization. Expert Systems with Applications, 36(2), 2400\u20132410. https:\/\/doi.org\/10.1016\/j.eswa.2007.12.052.","journal-title":"Expert Systems with Applications"},{"issue":"2","key":"3614_CR22","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1007\/s11192-011-0600-x","volume":"91","author":"X Liu","year":"2012","unstructured":"Liu, X., Gl\u00e4nzel, W., & De Moor, B. (2012). Optimal and hierarchical clustering of large-scale hybrid networks for scientific mapping. Scientometrics, 91(2), 473\u2013493. https:\/\/doi.org\/10.1007\/s11192-011-0600-x.","journal-title":"Scientometrics"},{"issue":"6","key":"3614_CR23","doi-asserted-by":"publisher","first-page":"1105","DOI":"10.1002\/asi.21312","volume":"61","author":"X Liu","year":"2010","unstructured":"Liu, X., Yu, S., Janssens, F., Gl\u00e4nzel, W., Moreau, Y., & De Moor, B. (2010). Weighted hybrid clustering by combining text mining and bibliometrics on a large-scale journal database. Journal of the American Society for Information Science and Technology, 61(6), 1105\u20131119. https:\/\/doi.org\/10.1002\/asi.21312.","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"3614_CR24","doi-asserted-by":"publisher","unstructured":"Liu, X., Yu, S., Moreau, Y., De Moor, B., Gl\u00e4nzel, W., & Janssens, F. (2009). Hybrid clustering of text mining and bibliometrics applied to journal sets. In Proceedings of the 2009 SIAM international conference on data mining (pp. 49\u201360). Society for Industrial and Applied Mathematics. https:\/\/doi.org\/10.1137\/1.9781611972795.5.","DOI":"10.1137\/1.9781611972795.5"},{"issue":"3","key":"3614_CR25","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276\u2013282. https:\/\/doi.org\/10.11613\/BM.2012.031.","journal-title":"Biochemia Medica"},{"key":"3614_CR26","unstructured":"Moed, H. F. (2006). Bibliometric rankings of world universities. CWTS Report 2006-01."},{"issue":"3","key":"3614_CR27","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1016\/j.joi.2010.01.002","volume":"4","author":"HF Moed","year":"2010","unstructured":"Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265\u2013277.","journal-title":"Journal of Informetrics"},{"issue":"3","key":"3614_CR28","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1007\/s11192-014-1319-2","volume":"100","author":"LG Nichols","year":"2014","unstructured":"Nichols, L. G. (2014). A topic model approach to measuring interdisciplinarity at the National Science Foundation. Scientometrics, 100(3), 741\u2013754.","journal-title":"Scientometrics"},{"key":"3614_CR29","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1016\/b978-0-444-70537-2.50019-2","volume-title":"Handbook of quantitative studies of science and technology","author":"JEJ Oberski","year":"1988","unstructured":"Oberski, J. E. J. (1988). Some statistical aspects of co-citation cluster analysis and a judgement by physicists. In van Raan (Ed.), Handbook of quantitative studies of science and technology (pp. 431\u2013462). Amsterdam: Elsevier. https:\/\/doi.org\/10.1016\/b978-0-444-70537-2.50019-2."},{"issue":"2","key":"3614_CR30","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1002\/asi.22746","volume":"64","author":"FN Piro","year":"2013","unstructured":"Piro, F. N., Aksnes, D. W., & R\u00f8rstad, K. (2013). A macro analysis of productivity differences across fields: Challenges in the measurement of scientific publishing. Journal of the American Society for Information Science and Technology, 64(2), 307\u2013320.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"45","key":"3614_CR31","doi-asserted-by":"publisher","first-page":"17268","DOI":"10.1073\/pnas.0806977105","volume":"105","author":"F Radicchi","year":"2008","unstructured":"Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268\u201317272.","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"3614_CR32","unstructured":"Robitaille, J.-P., Macaluso, B., Pollitt, A., Gunashekar, S., & Larivi\u00e8re, V. (2015). Comparative scientometric assessment of the results of ERC-funded projects (Bibliometric assessment report (D5)). Retrieved from https:\/\/erc.europa.eu\/sites\/default\/files\/document\/file\/ERC_Bibliometrics_report.pdf."},{"key":"3614_CR33","doi-asserted-by":"publisher","unstructured":"Schumacher, J., Zazworka, N., Shull, F., Seaman, C., & Shaw, M. (2010). Building empirical support for automated code smell detection. In Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement (p. 8). ACM. https:\/\/doi.org\/10.1145\/1852786.1852797.","DOI":"10.1145\/1852786.1852797"},{"issue":"11","key":"3614_CR35","doi-asserted-by":"publisher","first-page":"1361","DOI":"10.1515\/CCLM.2009.306","volume":"47","author":"AM Simundic","year":"2009","unstructured":"Simundic, A. M., Nikolac, N., Ivankovic, V., Ferenec-Ruzic, D., Magdic, B., & Kvaternik, M. (2009). Comparison of visual vs. automated detection of lipemic, icteric and hemolyzed specimens: Can we rely on a human eye? Clinical Chemistry and Laboratory Medicine, 47(11), 1361\u20131365. https:\/\/doi.org\/10.1515\/CCLM.2009.306.","journal-title":"Clinical Chemistry and Laboratory Medicine"},{"issue":"4","key":"3614_CR37","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1002\/asi.4630240406","volume":"24","author":"H Small","year":"1973","unstructured":"Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for information Science, 24(4), 265\u2013269. https:\/\/doi.org\/10.1002\/asi.4630240406.","journal-title":"Journal of the American Society for information Science"},{"issue":"4","key":"3614_CR38","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","volume":"45","author":"M Sokolova","year":"2009","unstructured":"Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45(4), 427\u2013437. https:\/\/doi.org\/10.1016\/j.ipm.2009.03.002.","journal-title":"Information Processing and Management"},{"key":"3614_CR39","doi-asserted-by":"publisher","unstructured":"Wei, X., & Croft, W. B. (2006). LDA-based document models for ad-hoc retrieval. In Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (pp. 178\u2013185). ACM. https:\/\/doi.org\/10.1145\/1148170.1148204.","DOI":"10.1145\/1148170.1148204"},{"key":"3614_CR40","doi-asserted-by":"publisher","unstructured":"Weismayer, C., Pezenka, I., & Gan, C. H. K. (2018). Aspect-based sentiment detection: Comparing human versus automated classifications of TripAdvisor reviews. In Information and communication technologies in tourism 2018 (pp. 365\u2013380). Springer, Cham. https:\/\/doi.org\/10.1007\/978-3-319-72923-7_28.","DOI":"10.1007\/978-3-319-72923-7_28"},{"issue":"3","key":"3614_CR41","doi-asserted-by":"publisher","first-page":"767","DOI":"10.1007\/s11192-014-1321-8","volume":"100","author":"CK Yau","year":"2014","unstructured":"Yau, C. K., Porter, A., Newman, N., & Suominen, A. (2014). Clustering scientific documents with topic modeling. Scientometrics, 100(3), 767\u2013786. https:\/\/doi.org\/10.1007\/s11192-014-1321-8.","journal-title":"Scientometrics"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03614-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-020-03614-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03614-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,18]],"date-time":"2021-07-18T00:07:26Z","timestamp":1626566846000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-020-03614-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,18]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["3614"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03614-2","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,18]]},"assertion":[{"value":"16 December 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 July 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}