{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,3]],"date-time":"2026-05-03T03:18:17Z","timestamp":1777778297692,"version":"3.51.4"},"reference-count":97,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2021,8,13]],"date-time":"2021-08-13T00:00:00Z","timestamp":1628812800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Information Visualization"],"published-print":{"date-parts":[[2021,10]]},"abstract":"<jats:p>In general, Natural Language Processing (NLP) algorithms exhibit black-box behavior. Users input text and output are provided with no explanation of how the results are obtained. In order to increase understanding and trust, users value transparent processing which may explain derived results and enable understanding of the underlying routines. Many approaches take an opaque approach by default when designing NLP tools and do not incorporate a means to steer and manipulate the intermediate NLP steps. We present an interactive, customizable, visual framework that enables users to observe and participate in the NLP pipeline processes, explicitly manipulate the parameters of each step, and explore the result visually based on user preferences. The visible NLP (VNLP) pipeline design is then applied to a text similarity application to demonstrate the utility and advantages of a visible and transparent NLP pipeline in supporting users to understand and justify both the process and results. We also report feedback on our framework from a modern languages expert.<\/jats:p>","DOI":"10.1177\/14738716211038898","type":"journal-article","created":{"date-parts":[[2021,8,14]],"date-time":"2021-08-14T02:10:57Z","timestamp":1628907057000},"page":"245-262","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":8,"title":["VNLP: Visible natural language processing"],"prefix":"10.1177","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5359-7056","authenticated-orcid":false,"given":"Mohammad","family":"Alharbi","sequence":"first","affiliation":[{"name":"Swansea University, Swansea, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Roach","sequence":"additional","affiliation":[{"name":"Swansea University, Swansea, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tom","family":"Cheesman","sequence":"additional","affiliation":[{"name":"Swansea University, Swansea, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert S","family":"Laramee","sequence":"additional","affiliation":[{"name":"Swansea University, Swansea, UK"},{"name":"University of Nottingham, Nottingham, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2021,8,13]]},"reference":[{"key":"bibr1-14738716211038898","author":"Smilkov D","year":"2017","journal-title":"ArXiv"},{"key":"bibr2-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744158"},{"key":"bibr3-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1177\/1473871620904671"},{"key":"bibr4-14738716211038898","unstructured":"Black Box. Meaning in the Cambridge English dictionary. https:\/\/dictionary.cambridge.org\/dictionary\/english\/black\u2013box (2021, accessed 14 May 2021)."},{"key":"bibr5-14738716211038898","unstructured":"Black Box. Definition of Black Box by Merriam-Webster. https:\/\/www.merriam-webster.com\/dictionary\/blackbox (2021, accessed 14 May 2021)."},{"key":"bibr6-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1145\/3236009"},{"key":"bibr7-14738716211038898","volume-title":"Proceedings 3rd workshop on visualization for the digital humanities (VIS4DH)","author":"Van Den Berg H"},{"key":"bibr8-14738716211038898","first-page":"336","volume-title":"Proceedings 1996 IEEE symposium on visual languages","author":"Shneiderman B"},{"issue":"3","key":"bibr9-14738716211038898","first-page":"62","volume":"23","author":"Feldman S.","year":"1999","journal-title":"Online"},{"key":"bibr10-14738716211038898","first-page":"117","volume-title":"IEEE pacific visualization symposium (PacificVis)","author":"Kucher K"},{"key":"bibr11-14738716211038898","doi-asserted-by":"publisher","DOI":"10.2991\/978-94-6239-186-4_2"},{"key":"bibr12-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2610422"},{"key":"bibr13-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13417"},{"key":"bibr14-14738716211038898","doi-asserted-by":"publisher","DOI":"10.3390\/computers8010017"},{"key":"bibr15-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1002\/widm.1071"},{"key":"bibr16-14738716211038898","first-page":"125","volume-title":"Eurographics conference on visualization (EuroVis) \u2013 STARs","author":"Wanner F"},{"key":"bibr17-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2834341"},{"key":"bibr18-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12873"},{"key":"bibr19-14738716211038898","unstructured":"Culy C. Free visualization tools software. http:\/\/www.chrisculy.net\/lx\/software\/ (2016, accessed 16 May 2021)."},{"key":"bibr20-14738716211038898","author":"Gunning D.","year":"2017","journal-title":"Defense Advanced Research Projects Agency (DARPA)"},{"key":"bibr21-14738716211038898","doi-asserted-by":"publisher","DOI":"10.23919\/MIPRO.2018.8400040"},{"key":"bibr22-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2870052"},{"key":"bibr23-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2843369"},{"key":"bibr24-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13092"},{"key":"bibr25-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2018.042731661"},{"key":"bibr26-14738716211038898","unstructured":"Smilkov D, Thorat N, Nicholson C, et al. Embedding projector: interactive visualization and interpretation of embeddings. arXiv:1611.05469. 2016."},{"key":"bibr27-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13672"},{"key":"bibr28-14738716211038898","first-page":"5686","volume-title":"Proceedings of the 2016 CHI conference on human factors in computing systems, CHI \u201816","author":"Krause J"},{"key":"bibr29-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1016\/j.tig.2020.03.005"},{"issue":"1","key":"bibr30-14738716211038898","first-page":"1064","volume":"26","author":"Spinner T","year":"2020","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"bibr31-14738716211038898","doi-asserted-by":"crossref","unstructured":"Tenney I, Wexler J, Bastings J, et al. The language interpretability tool: extensible, interactive visualizations and analysis for nlp models. arXiv:2008.05122. 2020.","DOI":"10.18653\/v1\/2020.emnlp-demos.15"},{"key":"bibr32-14738716211038898","first-page":"1","volume-title":"Proceedings of the 58th annual meeting of the association for computational linguistics: tutorial abstracts","author":"Belinkov Y"},{"key":"bibr33-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864499"},{"key":"bibr34-14738716211038898","first-page":"4902","volume-title":"Proceedings of the 58th annual meeting of the association for computational linguistics","author":"Ribeiro MT"},{"key":"bibr35-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12798"},{"key":"bibr36-14738716211038898","first-page":"127","volume-title":"IEEE conference on visual analytics science and technology (VAST)","author":"J\u00e4nicke S"},{"key":"bibr37-14738716211038898","first-page":"1","volume-title":"24th international conference on information visualisation","author":"Alharbi M"},{"key":"bibr38-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/j.1432-1033.1970.tb01046.x"},{"key":"bibr39-14738716211038898","first-page":"32","volume-title":"Proceedings of the NoDaLiDa 2017 workshop processing historical language","author":"Sch\u00e4tzle C"},{"issue":"2","key":"bibr40-14738716211038898","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1080\/10618600.1993.10474605","volume":"2","author":"Church KW","year":"1993","journal-title":"J Comput Graph Stat"},{"key":"bibr41-14738716211038898","first-page":"59","volume-title":"International conference on information visualization theory and applications (IVAPP)","author":"J\u00e4nicke S"},{"key":"bibr42-14738716211038898","volume-title":"Proceedings of the workshop on visualization for the digital humanities","author":"Silvia S"},{"key":"bibr43-14738716211038898","first-page":"i83","volume":"30","author":"J\u00e4nicke S","year":"2015","journal-title":"Digit Scholarsh Human"},{"key":"bibr44-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2008.172"},{"key":"bibr45-14738716211038898","volume":"1","author":"Alharbi MS","year":"2020","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"bibr46-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12618"},{"key":"bibr47-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1177\/1473871613495845"},{"key":"bibr48-14738716211038898","first-page":"115","volume-title":"2007 IEEE symposium on visual analytics science and technology","author":"Keim DA"},{"key":"bibr49-14738716211038898","first-page":"361","volume-title":"Proceedings of the first IEEE conference on visualization: visualization\u201990","author":"Inselberg A"},{"key":"bibr50-14738716211038898","first-page":"91","volume-title":"2009 IEEE symposium on visual analytics science and technology","author":"Collins C"},{"key":"bibr51-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2010.194"},{"key":"bibr52-14738716211038898","doi-asserted-by":"crossref","unstructured":"Castell\u00e0 Q, Sutton C. Word STORMS. Multiples of word clouds for visual comparison of documents. In: WWW 2014 \u2013 Proceedings of the 23rd international conference on world wide web, Seoul, Korea, 7\u201311 April 2014. New York: ACM.","DOI":"10.1145\/2566486.2567977"},{"key":"bibr53-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2745859"},{"key":"bibr54-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346677"},{"key":"bibr55-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12376"},{"key":"bibr56-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2009.01439.x"},{"key":"bibr57-14738716211038898","volume-title":"WordNet","author":"Fellbaum C","year":"2012"},{"key":"bibr58-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2009.165"},{"key":"bibr59-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2016.2598590"},{"key":"bibr60-14738716211038898","first-page":"110","volume-title":"Proceedings \u2013 IEEE symposium on information visualization, INFO VIS","author":"Wattenberg M."},{"key":"bibr61-14738716211038898","first-page":"60","volume-title":"Proceedings of the 14th linguistic annotation workshop","author":"Beck C"},{"key":"bibr62-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66984-7_10"},{"key":"bibr63-14738716211038898","volume-title":"Proceedings of the the seventh ACM conference on hypertext, HYPERTEXT \u201896","author":"Salton G"},{"key":"bibr64-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809071"},{"key":"bibr65-14738716211038898","author":"Dictionary OE","year":"1989","journal-title":"Oxford references online"},{"key":"bibr66-14738716211038898","first-page":"222","volume-title":"Advances in information retrieval","author":"Makrehchi M"},{"key":"bibr67-14738716211038898","first-page":"1661","volume-title":"Proceedings of the international joint conference on neural networks","volume":"3","author":"Silva C"},{"key":"bibr68-14738716211038898","first-page":"1","volume-title":"2012 Nirma University international conference on engineering (NUiCONE)","author":"Joshi H"},{"key":"bibr69-14738716211038898","doi-asserted-by":"publisher","DOI":"10.12928\/telkomnika.v13i4.2389"},{"key":"bibr70-14738716211038898","unstructured":"Frakes WB. Stemming algorithms. Hoboken, NJ: Prentice-Hall, Inc, 1992. pp.131\u2013160."},{"key":"bibr71-14738716211038898","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1037"},{"issue":"1","key":"bibr72-14738716211038898","first-page":"22","volume":"11","author":"Lovins JB.","year":"1968","journal-title":"Mech Transl Comput Linguistics"},{"key":"bibr73-14738716211038898","volume-title":"A linguistic failure analysis of classification of medical publications: a study on stemming vs lemmatization","author":"Di Nunzio GM","year":"2019"},{"key":"bibr74-14738716211038898","first-page":"625","volume-title":"Proceedings of the thirteenth ACM international conference on information and knowledge management, CIKM \u201804","author":"Korenius T"},{"key":"bibr75-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2017.2744478"},{"key":"bibr76-14738716211038898","volume-title":"1st international conference on learning representations, ICLR 2013","author":"Mikolov T"},{"key":"bibr77-14738716211038898","unstructured":"Devlin J, Chang MW, Lee K, et al. Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2019."},{"key":"bibr78-14738716211038898","first-page":"957","volume-title":"International conference on machine learning","author":"Kusner M"},{"key":"bibr79-14738716211038898","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"issue":"3","key":"bibr80-14738716211038898","first-page":"129","volume":"27","author":"Robertson SE","year":"1976","journal-title":"J Assoc Inf Sci Technol"},{"key":"bibr81-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007186"},{"key":"bibr82-14738716211038898","first-page":"3111","volume-title":"Advances in neural information processing systems","author":"Mikolov T","year":"2013"},{"key":"bibr83-14738716211038898","volume-title":"Pattern recognition and machine learning","author":"Bishop CM.","year":"2006"},{"key":"bibr84-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4842-2388-8"},{"key":"bibr85-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1179\/000870403235002042"},{"key":"bibr86-14738716211038898","volume-title":"Computer graphics and visual computing (CGVC)","author":"Roberts RC"},{"key":"bibr87-14738716211038898","volume":"0205028","author":"Loper E","year":"2002","journal-title":"arXiv preprint cs\/"},{"key":"bibr88-14738716211038898","unstructured":"Cheesman T, Flanagan K, Thiel S. Translation array prototype 1: project overview. Technical report 1, The Arts and Humanities Research Council (AHRC), 2012."},{"key":"bibr89-14738716211038898","unstructured":"Konrad M. A lemmatizer for german language text. https:\/\/github.com\/WZBSocialScienceCenter\/germalemma (2017). (accessed 10 April 2020)."},{"key":"bibr90-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-017-2390-9_2"},{"key":"bibr91-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90021-0"},{"key":"bibr92-14738716211038898","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2015.2511718"},{"key":"bibr93-14738716211038898","unstructured":"Cheesman T, Flanagan K, Thiel S. Translation array prototype 1: project overview. Jan 2013. http:\/\/delightedbeauty.org\/ (accessed 7 June 2019)."},{"key":"bibr94-14738716211038898","unstructured":"Cheesman T. delightedbeauty.org. http:\/\/www.delightedbeauty.org\/ (2011, accessed 16 February 2017)."},{"issue":"4","key":"bibr95-14738716211038898","first-page":"739","volume":"32","author":"Cheesman T","year":"2017","journal-title":"Digit Scholarsh Human"},{"key":"bibr96-14738716211038898","doi-asserted-by":"crossref","unstructured":"Cheesman T, Roos A. Version variation visualization (VVV): case studies on the Hebrew Haggadah in English. J Data Min Digital Humanities 2017. Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages. DoI: https:\/\/doi.org\/10.46298\/jdmdh.3704","DOI":"10.46298\/jdmdh.3704"},{"key":"bibr97-14738716211038898","volume-title":"Eurographics 2020 - education papers","author":"Firat EE"}],"container-title":["Information Visualization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716211038898","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/14738716211038898","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/14738716211038898","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T19:19:10Z","timestamp":1777490350000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/14738716211038898"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,13]]},"references-count":97,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,10]]}},"alternative-id":["10.1177\/14738716211038898"],"URL":"https:\/\/doi.org\/10.1177\/14738716211038898","relation":{},"ISSN":["1473-8716","1473-8724"],"issn-type":[{"value":"1473-8716","type":"print"},{"value":"1473-8724","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,13]]}}}