{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,5]],"date-time":"2026-01-05T21:58:54Z","timestamp":1767650334972,"version":"3.44.0"},"reference-count":52,"publisher":"SAGE Publications","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2018,4,12]]},"DOI":"10.3233\/sw-170282","type":"journal-article","created":{"date-parts":[[2017,5,12]],"date-time":"2017-05-12T11:46:48Z","timestamp":1494589608000},"page":"355-379","source":"Crossref","is-referenced-by-count":1,"title":["An extended study of content and crowdsourcing-related performance factors in named entity annotation"],"prefix":"10.1177","volume":"9","author":[{"given":"Oluwaseyi","family":"Feyisetan","sequence":"first","affiliation":[{"name":"University of Southampton, Southampton, UK. E-mail:\u00a0oof1v13@soton.ac.uk"}]},{"given":"Elena","family":"Simperl","sequence":"additional","affiliation":[{"name":"University of Southampton, UK. E-mails:\u00a0e.simperl@soton.ac.uk,\u00a0r.tinati@soton.ac.uk"}]},{"given":"Markus","family":"Luczak-Roesch","sequence":"additional","affiliation":[{"name":"Victoria University of Wellington, New Zealand. E-mail:\u00a0markus.luczak-roesch@vuw.ac.nz"}]},{"given":"Ramine","family":"Tinati","sequence":"additional","affiliation":[{"name":"University of Southampton, UK. E-mails:\u00a0e.simperl@soton.ac.uk,\u00a0r.tinati@soton.ac.uk"}]},{"given":"Nigel","family":"Shadbolt","sequence":"additional","affiliation":[{"name":"University of Oxford, UK. E-mail:\u00a0nigel.shadbolt@jesus.ox.ac.uk"}]}],"member":"179","reference":[{"key":"10.3233\/SW-170282_ref1","unstructured":"L.\u00a0Aroyo and C.\u00a0Welty, Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard, in: Proceedings of the 3rd Annual ACM Web Science Conference, WebSci\u201913, Paris, France, 2013, H.\u00a0Davis, H.\u00a0Halpin, A.\u00a0Pentland, M.\u00a0Bernstein, L.\u00a0Adamic, H.\u00a0Alani, A.\u00a0Monnin and R.\u00a0Rogers, eds, ACM, 2013."},{"issue":"1","key":"10.3233\/SW-170282_ref2","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1609\/aimag.v36i1.2564","article-title":"Truth is a lie: Crowd truth and the seven myths of human annotation","volume":"36","author":"Aroyo","year":"2015","journal-title":"AI Magazine"},{"key":"10.3233\/SW-170282_ref3","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-76298-0_52"},{"key":"10.3233\/SW-170282_ref4","unstructured":"A.E.C.\u00a0Basave, A.\u00a0Varga, M.\u00a0Rowe, M.\u00a0Stankovic and A.-S.\u00a0Dadzie, Making sense of microposts (#MSM2013) concept extraction challenge, in: Proceedings of the Concept Extraction Challenge at the Workshop on \u2018Making Sense of Microposts\u2019, Rio de Janeiro, Brazil, May 13, 2013, A.E.\u00a0Cano, M.\u00a0Rowe, M.\u00a0Stankovic and A.-S.\u00a0Dadzie, eds, CEUR Workshop Proceedings, Vol.\u00a01019, CEUR-WS.org, 2013, pp.\u00a01\u201315, http:\/\/ceur-ws.org\/Vol-1019\/msm2013-challenge-report.pdf."},{"key":"10.3233\/SW-170282_ref5","doi-asserted-by":"crossref","unstructured":"P.K.\u00a0Bhowmick, A.\u00a0Basu and P.\u00a0Mitra, An agreement measure for determining inter-annotator reliability of human judgements on affective text, in: Coling 2008: Proceedings of the Workshop on Human Judgements in Computational Linguistics, Manchester, UK, 23 August 2008, R.\u00a0Artstein, G.\u00a0Boleda, F.\u00a0Keller and S.\u00a0Schulte im Walde, eds, Coling 2008 Organization Committee, 2008, pp.\u00a058\u201365, http:\/\/www.aclweb.org\/anthology\/W08-1209.","DOI":"10.3115\/1611628.1611637"},{"key":"10.3233\/SW-170282_ref6","doi-asserted-by":"crossref","unstructured":"K.\u00a0Bontcheva, I.\u00a0Roberts, L.\u00a0Derczynski and D.P.\u00a0Rout, The GATE crowdsourcing plugin: Crowdsourcing annotated corpora made easy, in: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, Gothenburg, Sweden, April 26\u201330, 2014, G.\u00a0Bouma and Y.\u00a0Parmentier, eds, Association for Computational Linguistics, 2014, pp.\u00a026\u201330, http:\/\/aclweb.org\/anthology\/E\/E14\/E14-2025.pdf.","DOI":"10.3115\/v1\/E14-2025"},{"key":"10.3233\/SW-170282_ref7","doi-asserted-by":"crossref","unstructured":"K.\u00a0Bontcheva, L.\u00a0Derczynski and I.\u00a0Roberts, Crowdsourcing named entity recognition and entity linking corpora, in: Handbook of Linguistic Annotation, N.\u00a0Ide and J.\u00a0Pustejovsky, eds, Springer, 2017, To appear.","DOI":"10.1007\/978-94-024-0881-2_32"},{"key":"10.3233\/SW-170282_ref8","unstructured":"K.\u00a0Braunschweig, M.\u00a0Thiele, J.\u00a0Eberius and W.\u00a0Lehner, Enhancing named entity extraction by effectively incorporating the crowd, in: Datenbanksysteme f\u00fcr Business, Technologie und Web (BTW),\u00a0\u2013 Workshopband, 15. Fachtagung des GI-Fachbereichs \u201cDatenbanken und Informationssysteme\u201d (DBIS). Proceedings, Magdeburg, Germany, 11.\u201315.3.2013, G.\u00a0Saake, A.\u00a0Henrich, W.\u00a0Lehner, T.\u00a0Neumann and V.\u00a0K\u00f6ppen, eds, LNI, Vol.\u00a0216, GI, 2013, pp.\u00a0181\u2013195, http:\/\/www.btw-2013.de\/proceedings\/Enhancing."},{"key":"10.3233\/SW-170282_ref9","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2507858"},{"key":"10.3233\/SW-170282_ref10","unstructured":"N.A.\u00a0Chinchor, Overview of MUC-7\/MET-2, in: Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Fairfax, Virginia, April 29\u2013May 1, 1998, NIST, 1998, http:\/\/www.itl.nist.gov\/iaui\/894.02\/related_projects\/muc\/proceedings\/muc_7_proceedings\/overview.html."},{"issue":"1","key":"10.3233\/SW-170282_ref11","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educational and Psychological Measurement"},{"key":"10.3233\/SW-170282_ref12","unstructured":"T.\u00a0Cohn and L.\u00a0Specia, Modelling annotator bias with multi-task Gaussian processes: An application to machine translation quality estimation, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, Vol.\u00a01: Long Papers, Sofia, Bulgaria, 4\u20139 August 2013, Association for Computational Linguistics, 2013, pp.\u00a032\u201342, http:\/\/aclweb.org\/anthology\/P\/P13\/P13-1004.pdf."},{"key":"10.3233\/SW-170282_ref13","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-2271-7"},{"key":"10.3233\/SW-170282_ref14","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"10.3233\/SW-170282_ref15","unstructured":"L.\u00a0Derczynski, K.\u00a0Bontcheva and I.\u00a0Roberts, Broad Twitter corpus: A diverse named entity recognition resource, in: COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, Osaka, Japan, December 11\u201316, 2016, N.\u00a0Calzolari, Y.\u00a0Matsumoto and R.\u00a0Prasad, eds, ACL, 2016, pp.\u00a01169\u20131179, http:\/\/aclweb.org\/anthology\/C\/C16\/C16-1111.pdf."},{"issue":"2","key":"10.3233\/SW-170282_ref16","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.ipm.2014.10.006","article-title":"Analysis of named entity recognition and linking for tweets","volume":"51","author":"Derczynski","year":"2015","journal-title":"Information Processing and Management"},{"key":"10.3233\/SW-170282_ref17","unstructured":"D.E.\u00a0Difallah, G.\u00a0Demartini and P.\u00a0Cudr\u00e9-Mauroux, Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms, in: Proceedings of the First International Workshop on Crowdsourcing Web Search, Lyon, France, April 17, 2012, R.A.\u00a0Baeza-Yates, S.\u00a0Ceri, P.\u00a0Fraternali and F.\u00a0Giunchiglia, eds, CEUR Workshop Proceedings, Vol.\u00a0842, CEUR-WS.org, 2012, pp.\u00a026\u201330, http:\/\/ceur-ws.org\/Vol-842\/crowdsearch-difallah.pdf."},{"key":"10.3233\/SW-170282_ref18","unstructured":"G.R.\u00a0Doddington, A.\u00a0Mitchell, M.A.\u00a0Przybocki, L.A.\u00a0Ramshaw, S.\u00a0Strassel and R.M.\u00a0Weischedel, The automatic content extraction (ACE) program\u00a0\u2013 tasks, data, and evaluation, in: Proceedings of the Fourth International Conference on Language Resources and Evaluation, LREC 2004, Lisbon, Portugal, May 26\u201328, 2004, European Language Resources Association, 2004, pp.\u00a026\u201328, http:\/\/www.lrec-conf.org\/proceedings\/lrec2004\/pdf\/5.pdf."},{"key":"10.3233\/SW-170282_ref19","doi-asserted-by":"publisher","DOI":"10.1145\/2660517.2660527"},{"key":"10.3233\/SW-170282_ref20","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741639"},{"key":"10.3233\/SW-170282_ref21","unstructured":"T.\u00a0Finin, W.\u00a0Murnane, A.\u00a0Karandikar, N.\u00a0Keller, J.\u00a0Martineau and M.\u00a0Dredze, Annotating named entities in Twitter data with crowdsourcing, in: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk, Los Angeles, June 2010, Association for Computational Linguistics, 2010, pp.\u00a080\u201388, http:\/\/www.aclweb.org\/anthology\/W10-0713."},{"key":"10.3233\/SW-170282_ref22","unstructured":"H.\u00a0Fromreide, D.\u00a0Hovy and A.\u00a0S\u00f8gaard, Crowdsourcing and annotating NER for Twitter #drift, in: Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26\u201331, 2014, N.\u00a0Calzolari, K.\u00a0Choukri, T.\u00a0Declerck, H.\u00a0Loftsson, B.\u00a0Maegaard, J.\u00a0Mariani, A.\u00a0Moreno, J.\u00a0Odijk and S.\u00a0Piperidis, eds, European Language Resources Association (ELRA), 2014, pp.\u00a02544\u20132547, http:\/\/www.lrec-conf.org\/proceedings\/lrec2014\/summaries\/421.html."},{"key":"10.3233\/SW-170282_ref24","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_11"},{"key":"10.3233\/SW-170282_ref25","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33290-6_23"},{"key":"10.3233\/SW-170282_ref26","unstructured":"N.\u00a0Lawson, K.\u00a0Eustice, M.\u00a0Perkowitz and M.\u00a0Yetisgen-Yildiz, Annotating large email datasets for named entity recognition with mechanical turk, in: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk, Los Angeles, June 2010, C.\u00a0Callison-Burch and M.\u00a0Dredze, eds, Association for Computational Linguistics, 2010, pp.\u00a071\u201379, http:\/\/www.aclweb.org\/anthology\/W10-0712."},{"key":"10.3233\/SW-170282_ref27","unstructured":"X.\u00a0Liu, S.\u00a0Zhang, F.\u00a0Wei and M.\u00a0Zhou, Recognizing named entities in tweets, in: The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, Portland, Oregon, USA, 19\u201324 June, 2011, D.\u00a0Lin, Y.\u00a0Matsumoto and R.\u00a0Mihalcea, eds, Association for Computational Linguistics, 2011, pp.\u00a0359\u2013367, http:\/\/www.aclweb.org\/anthology\/P11-1037."},{"key":"10.3233\/SW-170282_ref28","unstructured":"M.\u00a0Luczak-R\u00f6sch and R.\u00a0Heese, Linked data authoring for non-experts, in: Proceedings of the WWW2009 Workshop on Linked Data on the Web, LDOW 2009, Madrid, Spain, April 20, 2009, C.\u00a0Bizer, T.\u00a0Heath, T.\u00a0Berners-Lee and K.\u00a0Idehen, eds, CEUR Workshop Proceedings, Vol.\u00a0538, CEUR-WS.org, 2009, http:\/\/ceur-ws.org\/Vol-538\/ldow2009_paper4.pdf."},{"key":"10.3233\/SW-170282_ref29","unstructured":"M.\u00a0Marrero, S.\u00a0S\u00e1nchez-Cuadrado, J.\u00a0Morato and Y.\u00a0Andreadakis, Evaluation of named entity extraction systems, in: Advances in Computational Linguistics, A.\u00a0Gelbukh, ed., Research in Computing Science, Vol.\u00a041, Instituto Polit\u00e9cnico Nacional, 2009, pp.\u00a047\u201358, http:\/\/www.cicling.org\/2009\/RCS-41\/047-058.pdf."},{"key":"10.3233\/SW-170282_ref30","unstructured":"D.\u00a0Maynard, K.\u00a0Bontcheva and H.\u00a0Cunningham, Towards a semantic extraction of named entities, in: Proceedings of RANLP\u00a0\u2013 2003 (Recent Advances in Natural Language Processing, Borovets, Bulgaria, 10\u201312 September 2003, R.\u00a0Mitkov, ed., BAS, 2003, pp.\u00a010\u201312."},{"key":"10.3233\/SW-170282_ref31","doi-asserted-by":"crossref","unstructured":"E.\u00a0Minkov, R.C.\u00a0Wang and W.W.\u00a0Cohen, Extracting personal names from email: Applying named entity recognition to informal text, in: HLT\/EMNLP 2005, Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, Vancouver, British Columbia, Canada, 6\u20138 October 2005, Association for Computational Linguistics, 2005, pp.\u00a0443\u2013450, http:\/\/aclweb.org\/anthology\/H\/H05\/H05-1056.pdf.","DOI":"10.3115\/1220575.1220631"},{"key":"10.3233\/SW-170282_ref32","unstructured":"R.\u00a0Morris, Crowdsourcing workshop: The emergence of affective crowdsourcing, in: CHI 2011 Workshop on Crowdsourcing and Human Computation: Systems, Studies, and Platforms, May 8, 2011, M.\u00a0Bernstein, B.\u00a0Hartmann, E.H.\u00a0Chi, N.\u00a0Kittur, L.\u00a0Chilton and R.C.\u00a0Miller, eds, 2011, http:\/\/www.humancomputation.com\/crowdcamp\/chi2011\/papers\/morris.pdf."},{"issue":"1","key":"10.3233\/SW-170282_ref33","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1075\/li.30.1.03nad","article-title":"A survey of named entity recognition and classification","volume":"30","author":"Nadeau","year":"2007","journal-title":"Lingvistic\u00e6 Investigationes"},{"key":"10.3233\/SW-170282_ref34","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"10.3233\/SW-170282_ref35","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/E14-1078"},{"key":"10.3233\/SW-170282_ref36","unstructured":"R.\u00a0Ramanath, M.\u00a0Choudhury, K.\u00a0Bali and R.S.\u00a0Roy, Crowd prefers the middle path: A new IAA metric for crowdsourcing reveals turker biases in query segmentation, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, Vol.\u00a01: Long Papers, Sofia, Bulgaria, 4\u20139 August 2013, Association for Computational Linguistics, 2013, pp.\u00a01713\u20131722, http:\/\/aclweb.org\/anthology\/P\/P13\/P13-1168.pdf."},{"key":"10.3233\/SW-170282_ref37","unstructured":"A.\u00a0Ritter, S.\u00a0Clark, Mausam and O.\u00a0Etzioni, Named entity recognition in tweets: An experimental study, in: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, EMNLP 2011, A meeting of SIGDAT, a Special Interest Group of the ACL, John McIntyre Conference Centre, Edinburgh, UK, 27\u201331 July 2011, Association for Computational Linguistics, 2011, pp.\u00a01524\u20131534, http:\/\/www.aclweb.org\/anthology\/D11-1141."},{"key":"10.3233\/SW-170282_ref38","unstructured":"G.\u00a0Rizzo and R.\u00a0Troncy, NERD: Evaluating named entity recognition tools in the web of data, in: Proceedings of Workshop on Web Scale Knowldge Extraction (WEKEX\u201911) Co-located with the 10th International Semantic Web Conference (ISWC 2011), Bonn, Germany, October 24, 2011, J.\u00a0Fan and A.\u00a0Kalyanpur, eds, 2011, http:\/\/www.eurecom.fr\/publication\/3517."},{"issue":"3","key":"10.3233\/SW-170282_ref39","doi-asserted-by":"publisher","first-page":"14","DOI":"10.4018\/ijswis.2013070102","article-title":"Crowdsourced knowledge acquisition: Towards hybrid-genre workflows","volume":"9","author":"Sabou","year":"2013","journal-title":"International Journal on Semantic Web and Information Systems"},{"key":"10.3233\/SW-170282_ref40","unstructured":"M.\u00a0Sabou, K.\u00a0Bontcheva, L.\u00a0Derczynski and A.\u00a0Scharl, Corpus annotation through crowdsourcing: Towards best practice guidelines, in: Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26\u201331, 2014, N.\u00a0Calzolari, K.\u00a0Choukri, T.\u00a0Declerck, H.\u00a0Loftsson, B.\u00a0Maegaard, J.\u00a0Mariani, A.\u00a0Moreno, J.\u00a0Odijk and S.\u00a0Piperidis, eds, European Language Resources Association (ELRA), 2014, pp.\u00a0859\u2013866, http:\/\/www.lrec-conf.org\/proceedings\/lrec2014\/summaries\/497.html."},{"key":"10.3233\/SW-170282_ref41","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_32"},{"key":"10.3233\/SW-170282_ref42","doi-asserted-by":"publisher","DOI":"10.3115\/1119176.1119195"},{"key":"10.3233\/SW-170282_ref43","doi-asserted-by":"publisher","DOI":"10.2200\/S00460ED1V01Y201212WBE004"},{"key":"10.3233\/SW-170282_ref44","doi-asserted-by":"publisher","DOI":"10.1109\/iMac4s.2013.6526500"},{"key":"10.3233\/SW-170282_ref45","doi-asserted-by":"crossref","unstructured":"R.\u00a0Snow, B.\u00a0O\u2019Connor, D.\u00a0Jurafsky and A.Y.\u00a0Ng, Cheap and fast\u00a0\u2013 but is it good? Evaluating non-expert annotations for natural language tasks, in: 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, A Meeting of SIGDAT, a Special Interest Group of the ACL, Honolulu, Hawaii, USA, 25\u201327 October 2008, Association for Computational Linguistics, 2008, pp.\u00a0254\u2013263, http:\/\/www.aclweb.org\/anthology\/D08-1027.","DOI":"10.3115\/1613715.1613751"},{"key":"10.3233\/SW-170282_ref46","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544865"},{"key":"10.3233\/SW-170282_ref47","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11964-9_29"},{"key":"10.3233\/SW-170282_ref48","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567989"},{"issue":"8","key":"10.3233\/SW-170282_ref49","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1145\/1378704.1378719","article-title":"Designing games with a purpose","volume":"51","author":"von Ahn","year":"2008","journal-title":"Communications of the ACM (CACM)"},{"key":"10.3233\/SW-170282_ref50","unstructured":"R.\u00a0Voyer, V.\u00a0Nygaard, W.\u00a0Fitzgerald and H.\u00a0Copperman, A hybrid model for annotating named entity training corpora, in: Proceedings of the Fourth Linguistic Annotation Workshop, Uppsala, Sweden, July 2010, N.\u00a0Xue and M.\u00a0Poesio, eds, Association for Computational Linguistics, 2010, pp.\u00a0243\u2013246, http:\/\/www.aclweb.org\/anthology\/W10-1839."},{"issue":"1","key":"10.3233\/SW-170282_ref51","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/s10579-012-9176-1","article-title":"Perspectives on crowdsourcing annotations for natural language processing","volume":"47","author":"Wang","year":"2013","journal-title":"Language Resources and Evaluation"},{"key":"10.3233\/SW-170282_ref52","unstructured":"M.\u00a0Yetisgen-Yildiz, I.\u00a0Solti, F.\u00a0Xia and S.\u00a0Halgrim, Preliminary experiments with Amazon\u2019s mechanical turk for annotating medical named entities, in: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk, Los Angeles, June 2010, C.\u00a0Callison-Burch and M.\u00a0Dredze, eds, Association for Computational Linguistics, 2010, pp.\u00a0180\u2013183, http:\/\/www.aclweb.org\/anthology\/W10-0728."},{"key":"10.3233\/SW-170282_ref53","doi-asserted-by":"publisher","DOI":"10.1109\/PASSAT\/SocialCom.2011.203"}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-170282","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,26]],"date-time":"2025-08-26T11:02:27Z","timestamp":1756206147000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-170282"}},"subtitle":[],"editor":[{"given":"Marta","family":"Sabou","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Vienna, Austria"}]},{"given":"Lora","family":"Aroyo","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, Netherlands"}]},{"given":"Kalina","family":"Bontcheva","sequence":"additional","affiliation":[{"name":"University of Sheffield, UK"}]},{"given":"Alessandro","family":"Bozzon","sequence":"additional","affiliation":[{"name":"Technische Universiteit Delft, Netherlands"}]}],"short-title":[],"issued":{"date-parts":[[2018,4,12]]},"references-count":52,"journal-issue":{"issue":"3"},"URL":"https:\/\/doi.org\/10.3233\/sw-170282","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"type":"electronic","value":"2210-4968"},{"type":"print","value":"1570-0844"}],"subject":[],"published":{"date-parts":[[2018,4,12]]}}}