{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T04:42:00Z","timestamp":1779165720894,"version":"3.51.4"},"reference-count":59,"publisher":"SAGE Publications","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2021,3,9]]},"abstract":"<jats:p>The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization in the quality of annotations, going against the usual practice of employing a small number of annotators.<\/jats:p>","DOI":"10.3233\/sw-200415","type":"journal-article","created":{"date-parts":[[2020,12,9]],"date-time":"2020-12-09T00:34:48Z","timestamp":1607474088000},"page":"403-421","source":"Crossref","is-referenced-by-count":6,"title":["Empirical methodology for crowdsourcing ground truth"],"prefix":"10.1177","volume":"12","author":[{"given":"Anca","family":"Dumitrache","sequence":"first","affiliation":[{"name":"Vrije Universiteit Amsterdam, De Boelelaan 1081, Amsterdam, Netherlands. E-mails:\u00a0anca.dmtrch@gmail.com,\u00a0oana.inel@gmail.com,\u00a0l.m.aroyo@gmail.com,\u00a0cawelty@gmail.com"},{"name":"FD Mediagroep, Amsterdam, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Oana","family":"Inel","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, De Boelelaan 1081, Amsterdam, Netherlands. E-mails:\u00a0anca.dmtrch@gmail.com,\u00a0oana.inel@gmail.com,\u00a0l.m.aroyo@gmail.com,\u00a0cawelty@gmail.com"},{"name":"TU Delft, Delft, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Timmermans","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, De Boelelaan 1081, Amsterdam, Netherlands. E-mails:\u00a0anca.dmtrch@gmail.com,\u00a0oana.inel@gmail.com,\u00a0l.m.aroyo@gmail.com,\u00a0cawelty@gmail.com"},{"name":"IBM Center for Advanced Studies Benelux, Johan Huizingalaan 765, Amsterdam, Netherlands. E-mails:\u00a0b.timmermans@nl.ibm.com,\u00a0rhjsips@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carlos","family":"Ortiz","sequence":"additional","affiliation":[{"name":"Netherlands eScience Center, Amsterdam, Netherlands. E-mail:\u00a0c.martinez@esciencecenter.nl"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert-Jan","family":"Sips","sequence":"additional","affiliation":[{"name":"IBM Center for Advanced Studies Benelux, Johan Huizingalaan 765, Amsterdam, Netherlands. E-mails:\u00a0b.timmermans@nl.ibm.com,\u00a0rhjsips@gmail.com"},{"name":"myTomorrows, Amsterdam, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lora","family":"Aroyo","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, De Boelelaan 1081, Amsterdam, Netherlands. E-mails:\u00a0anca.dmtrch@gmail.com,\u00a0oana.inel@gmail.com,\u00a0l.m.aroyo@gmail.com,\u00a0cawelty@gmail.com"},{"name":"Google, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chris","family":"Welty","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, De Boelelaan 1081, Amsterdam, Netherlands. E-mails:\u00a0anca.dmtrch@gmail.com,\u00a0oana.inel@gmail.com,\u00a0l.m.aroyo@gmail.com,\u00a0cawelty@gmail.com"},{"name":"Google, New York, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/SW-200415_ref1","unstructured":"A.R.\u00a0Aronson, Effective mapping of biomedical text to the UMLS metathesaurus: The MetaMap program, in: Proceedings of the AMIA Symposium, American Medical Informatics Association, 2001, p.\u00a017, PMID: 11825149, https:\/\/www.ncbi.nlm.nih.gov\/pubmed\/11825149."},{"key":"10.3233\/SW-200415_ref2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v39i4.2834"},{"key":"10.3233\/SW-200415_ref3","unstructured":"L.\u00a0Aroyo and C.\u00a0Welty, Harnessing disagreement for event semantics, in: Proceedings of the 2nd International Workshop on Detection, Representation, and Exploitation of Events in the Semantic Web (DeRiVE 2012), 11th International Semantic Web Conference, 2012, p.\u00a031."},{"key":"10.3233\/SW-200415_ref4","unstructured":"L.\u00a0Aroyo and C.\u00a0Welty, Measuring crowd truth for medical relation extraction, in: AAAI 2013 Fall Symposium on Semantics for Big Data, 2013."},{"key":"10.3233\/SW-200415_ref5","unstructured":"L.\u00a0Aroyo and C.\u00a0Welty, Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard, in: Proceedings of the 5th Annual ACM Web Science Conference, 2013."},{"key":"10.3233\/SW-200415_ref6","doi-asserted-by":"publisher","first-page":"31","DOI":"10.15346\/hc.v1i1.3","article-title":"The three sides of CrowdTruth","volume":"1","author":"Aroyo","year":"2014","journal-title":"Journal of Human Computation"},{"issue":"1","key":"10.3233\/SW-200415_ref7","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1609\/aimag.v36i1.2564","article-title":"Truth is a lie: Crowd truth and the seven myths of human annotation","volume":"36","author":"Aroyo","year":"2015","journal-title":"AI Magazine"},{"issue":"4","key":"10.3233\/SW-200415_ref8","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1162\/COLI_a_00074","article-title":"What determines inter-coder agreement in manual annotations? A meta-analytic investigation","volume":"37","author":"Bayerl","year":"2011","journal-title":"Comput. Linguist."},{"key":"10.3233\/SW-200415_ref9","doi-asserted-by":"publisher","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): Integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic acids research"},{"key":"10.3233\/SW-200415_ref10","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488403"},{"key":"10.3233\/SW-200415_ref11","doi-asserted-by":"crossref","unstructured":"J.\u00a0Bragg, D.S.\u00a0Weld et al., Crowdsourcing multi-label classification for taxonomy creation, in: First AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2013, http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP13\/paper\/view\/7560.","DOI":"10.1609\/hcomp.v1i1.13091"},{"issue":"2","key":"10.3233\/SW-200415_ref12","first-page":"249","article-title":"Assessing agreement on classification tasks: The kappa statistic","volume":"22","author":"Carletta","year":"1996","journal-title":"Comput. Linguist."},{"key":"10.3233\/SW-200415_ref13","unstructured":"T.\u00a0Caselli, R.\u00a0Sprugnoli and O.\u00a0Inel, Temporal information annotation: Crowd vs. experts, in: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), N.C.C.\u00a0Chair, K.\u00a0Choukri, T.\u00a0Declerck, S.\u00a0Goggi, M.\u00a0Grobelnik, B.\u00a0Maegaard, J.\u00a0Mariani, H.\u00a0Mazo, A.\u00a0Moreno, J.\u00a0Odijk and S.\u00a0Piperidis, eds, European Language Resources Association (ELRA), 2016. ISBN 978-2-9517408-9-1."},{"key":"10.3233\/SW-200415_ref14","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-48472-3_13"},{"key":"10.3233\/SW-200415_ref15","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"10.3233\/SW-200415_ref16","doi-asserted-by":"crossref","unstructured":"N.\u00a0Chang, R.\u00a0Lee-Goldman and M.\u00a0Tseng, Linguistic wisdom from the crowd, in: Third AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016, https:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP15\/paper\/view\/11737.","DOI":"10.1609\/hcomp.v3i1.13266"},{"key":"10.3233\/SW-200415_ref17","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11915-1_3"},{"issue":"2","key":"10.3233\/SW-200415_ref19","doi-asserted-by":"publisher","first-page":"11:1","DOI":"10.1145\/3152889","article-title":"Crowdsourcing ground truth for medical relation extraction","volume":"8","author":"Dumitrache","year":"2018","journal-title":"ACM Transactions on Interactive Intelligent Systems (TiiS)"},{"key":"10.3233\/SW-200415_ref20","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/n15-1089"},{"key":"10.3233\/SW-200415_ref21","unstructured":"T.\u00a0Finin, W.\u00a0Murnane, A.\u00a0Karandikar, N.\u00a0Keller, J.\u00a0Martineau and M.\u00a0Dredze, Annotating named entities in Twitter data with crowdsourcing, in: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk, CSLDAMT \u201810, Association for Computational Linguistics, 2010, pp.\u00a080\u201388, http:\/\/dl.acm.org\/citation.cfm?id=1866696.1866709."},{"issue":"3","key":"10.3233\/SW-200415_ref22","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1080\/09298215.2016.1200631","article-title":"The problem of limited inter-rater agreement in modelling music similarity","volume":"45","author":"Flexer","year":"2016","journal-title":"Journal of New Music Research"},{"key":"10.3233\/SW-200415_ref23","unstructured":"F.\u00a0Font, J.\u00a0Serr\u00e0 and X.\u00a0Serra, Audio clip classification using social tags and the effect of tag expansion, in: Audio Engineering Society 53rd International Conference on Semantic Audio, Audio Engineering Society, 2014."},{"key":"10.3233\/SW-200415_ref24","unstructured":"O.\u00a0Inel, T.\u00a0Caselli and L.\u00a0Aroyo, Crowdsourcing salient information from news and tweets, in: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), N.C.C.\u00a0Chair, K.\u00a0Choukri, T.\u00a0Declerck, S.\u00a0Goggi, M.\u00a0Grobelnik, B.\u00a0Maegaard, J.\u00a0Mariani, H.\u00a0Mazo, A.\u00a0Moreno, J.\u00a0Odijk and S.\u00a0Piperidis, eds, European Language Resources Association (ELRA), 2016. ISBN 978-2-9517408-9-1."},{"key":"10.3233\/SW-200415_ref25","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11915-1_31"},{"key":"10.3233\/SW-200415_ref26","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"10.3233\/SW-200415_ref27","unstructured":"D.\u00a0Jurgens, Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels, in: HLT-NAACL, 2013, pp.\u00a0556\u2013562, https:\/\/www.aclweb.org\/anthology\/N13-1062."},{"key":"10.3233\/SW-200415_ref28","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-12-486"},{"key":"10.3233\/SW-200415_ref29","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"issue":"2","key":"10.3233\/SW-200415_ref30","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1007\/BF02769550","article-title":"On the definition of \u201cpicture\u201d","volume":"14","author":"Knowlton","year":"1966","journal-title":"AV Communication Review"},{"key":"10.3233\/SW-200415_ref31","doi-asserted-by":"publisher","DOI":"10.1109\/icde.2014.6816717"},{"key":"10.3233\/SW-200415_ref32","unstructured":"J.H.\u00a0Lau, A.\u00a0Clark and S.\u00a0Lappin, Measuring gradience in speakers\u2019 grammaticality judgements, in: Proceedings of the 36th Annual Conference of the Cognitive, Science Society, 2014, pp.\u00a0821\u2013826."},{"issue":"5","key":"10.3233\/SW-200415_ref33","doi-asserted-by":"publisher","first-page":"711","DOI":"10.1007\/s00778-013-0328-8","article-title":"Hybrid entity clustering using crowds and data","volume":"22","author":"Lee","year":"2013","journal-title":"The VLDB Journal"},{"key":"10.3233\/SW-200415_ref34","doi-asserted-by":"crossref","unstructured":"C.H.\u00a0Lin, D.S.\u00a0Weld et al., To Re (Label), or Not to Re (Label), in: Second AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2014, http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP14\/paper\/view\/8978.","DOI":"10.1609\/hcomp.v2i1.13167"},{"key":"10.3233\/SW-200415_ref35","doi-asserted-by":"crossref","unstructured":"A.\u00a0Mao, E.\u00a0Kamar, Y.\u00a0Chen, E.\u00a0Horvitz, M.E.\u00a0Schwamb, C.J.\u00a0Lintott and A.M.\u00a0Smith, Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing, in: First AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2013, https:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP13\/paper\/view\/7497.","DOI":"10.1609\/hcomp.v1i1.13075"},{"key":"10.3233\/SW-200415_ref36","doi-asserted-by":"crossref","unstructured":"T.\u00a0McDonnell, M.\u00a0Lease, M.\u00a0Kutlu and T.\u00a0Elsayed, Why is that relevant? Collecting annotator rationales for relevance judgments, in: Fourth AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016, http:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP16\/paper\/view\/14043.","DOI":"10.1609\/hcomp.v4i1.13287"},{"issue":"2","key":"10.3233\/SW-200415_ref37","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the sampling error of the difference between correlated proportions or percentages","volume":"12","author":"McNemar","year":"1947","journal-title":"Psychometrika"},{"key":"10.3233\/SW-200415_ref38","unstructured":"T.\u00a0Mikolov, I.\u00a0Sutskever, K.\u00a0Chen, G.\u00a0Corrado and J.\u00a0Dean, Distributed representations of words and phrases and their compositionality, in: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS\u201913, Vol.\u00a02, Curran Associates Inc., USA, 2013, pp.\u00a03111\u20133119, http:\/\/dl.acm.org\/citation.cfm?id=2999792.2999959."},{"key":"10.3233\/SW-200415_ref39","doi-asserted-by":"crossref","unstructured":"M.\u00a0Mintz, S.\u00a0Bills, R.\u00a0Snow and D.\u00a0Jurafsky, Distant supervision for relation extraction without labeled data, in: Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Vol.\u00a02, Association for Computational Linguistics, 2009, pp.\u00a01003\u20131011, https:\/\/www.aclweb.org\/anthology\/P09-1113.","DOI":"10.3115\/1690219.1690287"},{"key":"10.3233\/SW-200415_ref40","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"10.3233\/SW-200415_ref41","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464482"},{"key":"10.3233\/SW-200415_ref42","doi-asserted-by":"publisher","DOI":"10.1145\/2615569.2615644"},{"key":"10.3233\/SW-200415_ref43","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/p14-2083"},{"key":"10.3233\/SW-200415_ref44","doi-asserted-by":"publisher","DOI":"10.3115\/1608829.1608840"},{"issue":"7638","key":"10.3233\/SW-200415_ref45","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1038\/nature21054","article-title":"A solution to the single-question crowd wisdom problem","volume":"541","author":"Prelec","year":"2017","journal-title":"Nature"},{"key":"10.3233\/SW-200415_ref46","unstructured":"J.\u00a0Pustejovsky, P.\u00a0Hanks, R.\u00a0Sauri, A.\u00a0See, R.\u00a0Gaizauskas, A.\u00a0Setzer, D.\u00a0Radev, B.\u00a0Sundheim, D.\u00a0Day, L.\u00a0Ferro et al., The TimeBank corpus, in: Corpus Linguistics, 2003, p.\u00a040, http:\/\/ucrel.lancs.ac.uk\/publications\/cl2003\/papers\/pustejovsky.pdf."},{"issue":"1","key":"10.3233\/SW-200415_ref47","doi-asserted-by":"publisher","first-page":"3","DOI":"10.15346\/hc.v2i1.2","article-title":"Crowdsourcing and the semantic web: A research manifesto","volume":"2","author":"Sarasua","year":"2015","journal-title":"Human Computation (HCOMP)"},{"key":"10.3233\/SW-200415_ref48","unstructured":"M.\u00a0Schaekermann, E.\u00a0Law, A.C.\u00a0Williams and W.\u00a0Callaghan, Resolvable vs. irresolvable ambiguity: A new hybrid framework for dealing with uncertain ground truth, in: 1st Workshop on Human-Centered Machine Learning at SIGCHI 2016, 2016."},{"key":"10.3233\/SW-200415_ref49","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488489"},{"key":"10.3233\/SW-200415_ref50","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613751"},{"key":"10.3233\/SW-200415_ref51","unstructured":"G.\u00a0Sober\u00f3n, L.\u00a0Aroyo, C.\u00a0Welty, O.\u00a0Inel, H.\u00a0Lin and M.\u00a0Overmeen, Measuring crowd truth: Disagreement metrics combined with worker behavior filters, in: 1st International Workshop on Crowdsourcing the Semantic Web, 12th International Semantic Web Conference, 2013."},{"key":"10.3233\/SW-200415_ref52","unstructured":"E.\u00a0van Miltenburg, B.\u00a0Timmermans and L.\u00a0Aroyo, The VU sound corpus: Adding more fine-grained annotations to the freesound database, in: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), N.C.C.\u00a0Chair, K.\u00a0Choukri, T.\u00a0Declerck, S.\u00a0Goggi, M.\u00a0Grobelnik, B.\u00a0Maegaard, J.\u00a0Mariani, H.\u00a0Mazo, A.\u00a0Moreno, J.\u00a0Odijk and S.\u00a0Piperidis, eds, European Language Resources Association (ELRA), 2016. ISBN 978-2-9517408-9-1."},{"issue":"5","key":"10.3233\/SW-200415_ref53","doi-asserted-by":"publisher","first-page":"879","DOI":"10.1016\/j.jbi.2012.04.004","article-title":"The EU-ADR corpus: Annotated drugs, diseases, targets, and their relationships","volume":"45","author":"Van Mulligen","year":"2012","journal-title":"Journal of biomedical informatics"},{"key":"10.3233\/SW-200415_ref54","doi-asserted-by":"publisher","DOI":"10.1145\/1629911.1630023"},{"key":"10.3233\/SW-200415_ref55","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/p14-1078"},{"key":"10.3233\/SW-200415_ref56","unstructured":"P.\u00a0Welinder, S.\u00a0Branson, P.\u00a0Perona and S.J.\u00a0Belongie, The multidimensional wisdom of crowds, in: Advances in Neural Information Processing Systems, 2010, pp.\u00a02424\u20132432, http:\/\/papers.nips.cc\/paper\/4074-the-multidimensional-wisdom-of-crowds."},{"key":"10.3233\/SW-200415_ref57","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35173-0_15"},{"key":"10.3233\/SW-200415_ref58","unstructured":"K.\u00a0Werling, A.T.\u00a0Chaganty, P.S.\u00a0Liang and C.D.\u00a0Manning, On-the-job learning with Bayesian decision theory, in: Advances in Neural Information Processing Systems, 2015, pp.\u00a03465\u20133473."},{"key":"10.3233\/SW-200415_ref59","unstructured":"J.\u00a0Whitehill, T.-F.\u00a0Wu, J.\u00a0Bergsma, J.R.\u00a0Movellan and P.L.\u00a0Ruvolo, Whose vote should count more: Optimal integration of labels from labelers of unknown expertise, in: Advances in Neural Information Processing Systems 22, Y.\u00a0Bengio, D.\u00a0Schuurmans, J.D.\u00a0Lafferty, C.K.I.\u00a0Williams and A.\u00a0Culotta, eds, Curran Associates, Inc., 2009, pp.\u00a02035\u20132043, http:\/\/papers.nips.cc\/paper\/3644-whose-vote-should-count-more-optimal-integration-of- labels-from-labelers-of-unknown-expertise.pdf."},{"key":"10.3233\/SW-200415_ref60","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2426"}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-200415","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:25:11Z","timestamp":1777613111000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-200415"}},"subtitle":[],"editor":[{"given":"Marta","family":"Sabou","sequence":"additional","affiliation":[{"name":"Technical University of Vienna, Austria"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Lora","family":"Aroyo","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, Netherlands"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Kalina","family":"Bontcheva","sequence":"additional","affiliation":[{"name":"University of Sheffield, UK"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Alessandro","family":"Bozzon","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Netherlands"}],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,3,9]]},"references-count":59,"journal-issue":{"issue":"3"},"URL":"https:\/\/doi.org\/10.3233\/sw-200415","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"value":"2210-4968","type":"electronic"},{"value":"1570-0844","type":"print"}],"subject":[],"published":{"date-parts":[[2021,3,9]]}}}