{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T00:30:28Z","timestamp":1789950628713,"version":"4.0.1"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2017,6,28]],"date-time":"2017-06-28T00:00:00Z","timestamp":1498608000000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources & Evaluation"],"published-print":{"date-parts":[[2018,6]]},"DOI":"10.1007\/s10579-017-9395-6","type":"journal-article","created":{"date-parts":[[2017,6,28]],"date-time":"2017-06-28T10:15:41Z","timestamp":1498644941000},"page":"503-532","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Creation and evaluation of large keyphrase extraction collections with multiple opinions"],"prefix":"10.1007","volume":"52","author":[{"given":"Lucas","family":"Sterckx","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Demeester","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Johannes","family":"Deleu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chris","family":"Develder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2017,6,28]]},"reference":[{"key":"9395_CR1","doi-asserted-by":"crossref","unstructured":"Augenstein, I., Das, M., Riedel, S., Vikraman, L., & McCallum, A. (2017). SemEval 2017 Task 10: ScienceIE\u2014Extracting keyphrases and relations from scientific publications. ArXiv e-prints 1704.02853.","DOI":"10.18653\/v1\/S17-2091"},{"key":"9395_CR2","doi-asserted-by":"crossref","unstructured":"Bird, S. (2006). NLTK: The natural language toolkit. In Proceedings of the COLING\/ACL on Interactive Presentation Sessions, Association for Computational Linguistics, pp.\u00a069\u201372.","DOI":"10.3115\/1225403.1225421"},{"key":"9395_CR3","doi-asserted-by":"publisher","unstructured":"Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. JMLR, 3(4\u20135):993\u20131022, doi: 10.1162\/jmlr.2003.3.4-5.993 , http:\/\/www.crossref.org\/jmlr_DOI.html .","DOI":"10.1162\/jmlr.2003.3.4-5.993"},{"key":"9395_CR4","unstructured":"Bougouin, A., & Boudin, F. (2014). Topicrank: ordonnancement de sujets pour lextraction automatique de termes-cls. TAL, 55(1):45\u201369, http:\/\/www.atala.org\/IMG\/pdf\/2._Bougoin-TAL55-1.pdf ."},{"key":"9395_CR5","unstructured":"Bowman, J. (2003). Essential cataloguing. Facet Pub., https:\/\/books.google.be\/books?id=C-7gAAAAMAAJ ."},{"key":"9395_CR6","doi-asserted-by":"crossref","unstructured":"Brill, E. (1992). A simple rule-based part of speech tagger. In Proceedings of the Workshop on Speech and Natural Language, Association for Computational Linguistics, pp. 112\u2013116.","DOI":"10.3115\/1075527.1075553"},{"key":"9395_CR7","doi-asserted-by":"publisher","unstructured":"Bulgarov, F. A., & Caragea, C. (2015). A comparison of supervised keyphrase extraction models. In Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18\u201322, 2015\u2014Companion Volume, pp. 13\u201314, doi: 10.1145\/2740908.2742776 .","DOI":"10.1145\/2740908.2742776"},{"issue":"3","key":"9395_CR8","doi-asserted-by":"crossref","first-page":"27:1","DOI":"10.1145\/1961189.1961199","volume":"2","author":"CC Chang","year":"2011","unstructured":"Chang, C. C., & Lin, C. J. (2011). Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3), 27:1\u201327:27.","journal-title":"ACM Transactions on Intelligent Systems and Technology"},{"key":"9395_CR9","unstructured":"Chen, T., & Guestrin, C. (2016). Xgboost: A scalable tree boosting system. CoRR abs\/1603.02754, http:\/\/arxiv.org\/abs\/1603.02754 ."},{"key":"9395_CR10","unstructured":"D\u2019Avanzo, E., Magnini, B., & Vallin, A. (2004). Keyphrase extraction for summarization purposes: The LAKE system at DUC-2004. In Proceedings of the 2004 DUC."},{"key":"9395_CR11","doi-asserted-by":"publisher","unstructured":"Elkan, C., & Noto, K. (2008). Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24\u201327, 2008, pp. 213\u2013220, doi: 10.1145\/1401890.1401920 .","DOI":"10.1145\/1401890.1401920"},{"issue":"5","key":"9395_CR12","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378.","journal-title":"Psychological Bulletin"},{"key":"9395_CR13","unstructured":"Frank, E., Paynter, G.W., Witten, I.H., Gutwin, C., Nevill-manning, C.G. (1999). Domain specific keyphrase extraction. In Proceedings of the 16th International Joint Conference on AI, pp. 668\u2013673."},{"issue":"2\u20133","key":"9395_CR14","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1179\/174327909X441090","volume":"34","author":"L Gazendam","year":"2009","unstructured":"Gazendam, L., Wartena, C., Malais, V., Schreiber, G., de Jong, A., & Brugman, H. (2009). Automatic annotation suggestions for audiovisual archives: Evaluation aspects. Interdisciplinary Science Reviews, 34(2\u20133), 172\u2013188. doi: 10.1179\/174327909X441090 .","journal-title":"Interdisciplinary Science Reviews"},{"key":"9395_CR15","doi-asserted-by":"publisher","unstructured":"Gazendam, L., Wartena, C., & Brussee, R. (2010). Thesaurus based term ranking for keyword extraction. In Database and Expert Systems Applications, DEXA, International Workshops, Bilbao, Spain, August 30\u2013September 3, 2010, pp. 9\u201353, doi: 10.1109\/DEXA.2010.31 .","DOI":"10.1109\/DEXA.2010.31"},{"key":"9395_CR16","unstructured":"Grineva, M., Grinev, M., & Lizorkin, D. (2009). Extracting key terms from noisy and multitheme documents. WWW 2009 MADRID! Track: Semantic\/Data Web \/ Session: Mining for Semantics, pp. 661\u2013670, http:\/\/dl.acm.org\/citation.cfm?id=1526798 ."},{"key":"9395_CR17","doi-asserted-by":"crossref","unstructured":"Hammouda, K. M., Matute, D. N., & Kamel, M. S. (2005). Corephrase: Keyphrase extraction for document clustering. In P. Perner & A. Imiya (Eds.), Machine learning and data mining in pattern recognition (pp. 265\u2013274). Springer: Berlin.","DOI":"10.1007\/11510888_26"},{"key":"9395_CR18","doi-asserted-by":"crossref","unstructured":"Hasan, K. S., & Ng, V. (2014). Automatic keyphrase extraction: A survey of the state of the art. In Proceedings of the Association for Computational Linguistics (ACL), Baltimore, Maryland: Association for Computational Linguistics.","DOI":"10.3115\/v1\/P14-1119"},{"key":"9395_CR19","unstructured":"Hulth, A. (2003). Improved automatic keyword extraction given more linguistic knowledge. In Proceedings of the 2003 Conference on Empirical Natural language Processing (2000). http:\/\/dl.acm.org\/citation.cfm?id=1119383 ."},{"key":"9395_CR20","doi-asserted-by":"crossref","unstructured":"Jiang, X., Hu, Y., & Li, H. (2009). A ranking approach to keyphrase extraction. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp. 756\u2013757.","DOI":"10.1145\/1571941.1572113"},{"key":"9395_CR21","doi-asserted-by":"crossref","unstructured":"Kim, S. N. & Kan, M. Y. (2009). Re-examining automatic keyphrase extraction approaches in scientific articles. In Proceedings of the Workshop on Multiword Expressions: Identification, Interpretation, Disambiguation and Applications, Association for Computational Linguistics, pp. 9\u201316.","DOI":"10.3115\/1698239.1698242"},{"key":"9395_CR22","unstructured":"Lievens, B., Baccarne, B., Veeckman, C., Logghe, S., & Schuurman, D. (2014). Drivers for end-users\u2019 collaboration in participatory innovation development and living lab processes. In 17th ACM Conference on Computer Supported Cooperative Work and Social Computing."},{"key":"9395_CR23","unstructured":"Liu, Z., Huang, W., Zheng, Y., & Sun, M. (2010). Automatic keyphrase extraction via topic decomposition. In Proceedings of the 2010 Conference on EMNLP, pp. 366\u2013376."},{"key":"9395_CR24","unstructured":"Lopez, P., & Romary, L. (2010). Humb: Automatic key term extraction from scientific articles in grobid. In Proceedings of the 5th International Workshop on Semantic Evaluation, Association for Computational Linguistics, pp. 248\u2013251."},{"key":"9395_CR25","unstructured":"Medelyan, O., & Witten, I. (2002). Thesaurus based automatic keyphrase indexing. In Proceedings of the 6th ACM\/IEED-CS Joint Conference on Digital Libraries, pp.\u00a0296\u2013297."},{"key":"9395_CR26","unstructured":"Mihalcea, R., & Csomai, A. (2007). Wikify!: Linking documents to encyclopedic knowledge. CIKM07, November 68, 2007, Lisboa, Portugal (July). http:\/\/dl.acm.org\/citation.cfm?id=1321475 ."},{"key":"9395_CR27","unstructured":"Mihalcea, R., & Tarau, P. (2004). TextRank: Bringing order into texts. In Proceedings of the 2004 Conference on EMNLP. http:\/\/acl.ldc.upenn.edu\/acl2004\/emnlp\/pdf\/Mihalcea.pdf ."},{"key":"9395_CR28","unstructured":"Park, Y., Byrd, R.J., & Boguraev, B. (2002). Automatic glossary extraction: Beyond terminology identification. In 19th International Conference on Computational Linguistics, COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24\u2013September 1, 2002. http:\/\/aclweb.org\/anthology\/C02-1142 ."},{"issue":"5","key":"9395_CR29","doi-asserted-by":"crossref","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","volume":"24","author":"G Salton","year":"1988","unstructured":"Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513\u2013523.","journal-title":"Information Processing & Management"},{"key":"9395_CR30","doi-asserted-by":"crossref","unstructured":"Sterckx, L., Demeester, T., Deleu, J., & Develder, C. (2015a). Topical word importance for fast keyphrase extraction. In Proceedings of the 24th International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee, pp. 121\u2013122.","DOI":"10.1145\/2740908.2742730"},{"key":"9395_CR31","doi-asserted-by":"crossref","unstructured":"Sterckx, L., Demeester, T., Deleu, J., & Develder, C. (2015b). When topic models disagree: Keyphrase extraction with multiple topic models. In Proceedings of the 24th International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee. pp. 123\u2013124.","DOI":"10.1145\/2740908.2742731"},{"key":"9395_CR32","doi-asserted-by":"crossref","unstructured":"Sterckx, L., Caragea, C., Demeester, T., & Develder, C. (2016). Supervised keyphrase extraction as positive unlabeled learning. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, November 2\u20134, 2016, Austin, Texas.","DOI":"10.18653\/v1\/D16-1198"},{"key":"9395_CR33","unstructured":"Turney, P. (1999). Learning to extract keyphrases from text. http:\/\/nparc.cisti-icist.nrc-cnrc.gc.ca\/npsi\/ctrl?action=rtdoc&an=8913245 ."},{"key":"9395_CR34","unstructured":"Turney, P. (2000). Learning algorithms for keyphrase extraction. Information Retrieval http:\/\/nparc.cisti-icist.nrc-cnrc.gc.ca\/npsi\/ctrl?action=rtdoc&an=8913713 ."},{"key":"9395_CR35","unstructured":"Wan, X., & Xiao, J. (2008). Single document keyphrase extraction using neighborhood knowledge. In Proceedings of the 23rd National Conference on Artificial Intelligence\u2014Volume 2, AAAI 2008. pp.\u00a0855\u2013860. http:\/\/dl.acm.org\/citation.cfm?id=1620163.1620205 ."},{"key":"9395_CR36","doi-asserted-by":"publisher","unstructured":"Wartena, C., Brussee, R., & Slakhorst, W. (2010). Keyword extraction using word co-occurrence. In 2010 Workshops on Database and Expert Systems Applications, pp. 54\u201358. doi: 10.1109\/DEXA.2010.32 , http:\/\/ieeexplore.ieee.org\/lpdocs\/epic03\/wrapper.htm?arnumber=5592000 .","DOI":"10.1109\/DEXA.2010.32"},{"key":"9395_CR37","unstructured":"Witten, I., Paynter, G., & Frank, E. (1999). KEA: Practical automatic keyphrase extraction. In Proceedings of the Fourth ACM Conference on Digital Libraries. http:\/\/dl.acm.org\/citation.cfm?id=313437 ."},{"issue":"1","key":"9395_CR38","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/4235.585893","volume":"1","author":"DH Wolpert","year":"1997","unstructured":"Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67\u201382.","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"9395_CR39","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1145\/1135777.1135813","volume":"2006","author":"Wt Yih","year":"2006","unstructured":"Yih, Wt., Goodman, J., & Carvalho, V. R. (2006). Finding advertising keywords on web pages. WWW, 2006, 213\u2013222.","journal-title":"WWW"},{"key":"9395_CR40","doi-asserted-by":"publisher","unstructured":"Zhang, Y., Zincir-Heywood, N., & Milios, E. (2005). Narrative text classification for automatic key phrase extraction in web document corpora. In Proceedings of the 7th Annual ACM International Workshop on Web Information and Data Management, ACM, New York, NY, USA, WIDM \u201905, pp.\u00a051\u201358. doi: 10.1145\/1097047.1097059 .","DOI":"10.1145\/1097047.1097059"},{"key":"9395_CR41","unstructured":"Zhao, W. X., Jiang, J., He, J., Song, Y., Achananuparp, P., Lim, E. P., & Li, X. (2011). Topical keyphrase extraction from twitter. In Proceedings of the 49th Annual Meeting of the ACL: HLT- Volume 1, Stroudsburg, PA, USA, HLT \u201911, pp. 379\u2013388. http:\/\/dl.acm.org\/citation.cfm?id=2002472.2002521 ."}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10579-017-9395-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-017-9395-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-017-9395-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,9,27]],"date-time":"2019-09-27T09:02:13Z","timestamp":1569574933000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10579-017-9395-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,28]]},"references-count":41,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,6]]}},"alternative-id":["9395"],"URL":"https:\/\/doi.org\/10.1007\/s10579-017-9395-6","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,6,28]]}}}