{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,29]],"date-time":"2025-10-29T03:42:34Z","timestamp":1761709354673,"version":"3.37.3"},"reference-count":26,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2017,7,22]],"date-time":"2017-07-22T00:00:00Z","timestamp":1500681600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["5R01LM010964"],"award-info":[{"award-number":["5R01LM010964"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2018,1,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The gap between domain experts and natural language processing expertise is a barrier to extracting understanding from clinical text. We describe a prototype tool for interactive review and revision of natural language processing models of binary concepts extracted from clinical notes. We evaluated our prototype in a user study involving 9 physicians, who used our tool to build and revise models for 2 colonoscopy quality variables. We report changes in performance relative to the quantity of feedback. Using initial training sets as small as 10 documents, expert review led to final F1scores for the \u201cappendiceal-orifice\u201d variable between 0.78 and 0.91 (with improvements ranging from 13.26% to 29.90%). F1for \u201cbiopsy\u201d ranged between 0.88 and 0.94 (\u22121.52% to 11.74% improvements). The average System Usability Scale score was 70.56. Subjective feedback also suggests possible design improvements.<\/jats:p>","DOI":"10.1093\/jamia\/ocx070","type":"journal-article","created":{"date-parts":[[2017,6,10]],"date-time":"2017-06-10T01:30:56Z","timestamp":1497058256000},"page":"81-87","source":"Crossref","is-referenced-by-count":29,"title":["NLPReViz: an interactive tool for natural language processing on clinical text"],"prefix":"10.1093","volume":"25","author":[{"given":"Gaurav","family":"Trivedi","sequence":"first","affiliation":[{"name":"Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Phuong","family":"Pham","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wendy W","family":"Chapman","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rebecca","family":"Hwa","sequence":"additional","affiliation":[{"name":"Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA"},{"name":"Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Janyce","family":"Wiebe","sequence":"additional","affiliation":[{"name":"Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA"},{"name":"Department of Computer Science, University of Pittsburgh, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Harry","family":"Hochheiser","sequence":"additional","affiliation":[{"name":"Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA"},{"name":"Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2017,7,22]]},"reference":[{"volume-title":"The Computer-based Patient Record: an Essential Technology for Health Care","year":"1997","author":"Dick","key":"2020110612460725900_ocx070-B1"},{"key":"2020110612460725900_ocx070-B2","first-page":"447","article-title":"Canary: An NLP platform for clinicians and researchers","volume":"2","author":"Malmasi","year":"2017","journal-title":"Appl Clin Inform."},{"key":"2020110612460725900_ocx070-B3","doi-asserted-by":"crossref","DOI":"10.1007\/0-387-36278-9_8","article-title":"Natural language processing in biomedicine","volume-title":"Biomedical Informatics: Computer Applications in Health Care and Biomedicine","author":"Friedman"},{"key":"2020110612460725900_ocx070-B4","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1007\/978-3-642-39146-0_2","article-title":"Combining HCI, natural language processing, and knowledge discovery-potential of IBM content analytics as an assistive technology in the biomedical field","volume-title":"Human-Computer Interaction and Knowledge Discovery in Complex, Unstructured, Big Data","author":"Holzinger","year":"2013"},{"key":"2020110612460725900_ocx070-B5","article-title":"Effective end-user interaction with machine learning","author":"Amershi","year":"2011","journal-title":"Proc Twenty-Fifth AAAI Conf Artificial Intell"},{"issue":"4","key":"2020110612460725900_ocx070-B6","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1609\/aimag.v35i4.2513","article-title":"Power to the people: the role of humans in interactive machine learning","volume":"35","author":"Amershi","year":"2014","journal-title":"AI Magazine."},{"key":"2020110612460725900_ocx070-B7","doi-asserted-by":"crossref","DOI":"10.1145\/1978942.1978967","article-title":"Apolo: making sense of large network data by combining rich user interaction and machine learning","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Chau"},{"issue":"12","key":"2020110612460725900_ocx070-B8","doi-asserted-by":"crossref","first-page":"2839","DOI":"10.1109\/TVCG.2012.277","article-title":"Visual classifier training for text document retrieval","volume":"18","author":"Heimerl","year":"2012","journal-title":"IEEE Transactions on Visualization and Computer Graphics."},{"key":"2020110612460725900_ocx070-B9","doi-asserted-by":"crossref","DOI":"10.1145\/604045.604056","article-title":"Interactive machine learning","volume-title":"Proceedings of the 8th International Conference on Intelligent User Interfaces","author":"Fails"},{"key":"2020110612460725900_ocx070-B10","doi-asserted-by":"crossref","DOI":"10.1145\/1978942.1978965","article-title":"Human model evaluation in interactive supervised learning","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Fiebrink"},{"issue":"4","key":"2020110612460725900_ocx070-B11","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1007\/s40708-016-0043-5","article-title":"Visual analytics for concept exploration in subspaces of patient groups","volume":"3","author":"Hund","year":"2016","journal-title":"Brain Inform."},{"key":"2020110612460725900_ocx070-B12","doi-asserted-by":"crossref","DOI":"10.1145\/2110363.2110464","article-title":"Deploying an interactive machine learning system in an evidence-based practice center: abstrackr","volume-title":"Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium","author":"Wallace"},{"issue":"6","key":"2020110612460725900_ocx070-B13","doi-asserted-by":"crossref","first-page":"1221","DOI":"10.1109\/TVCG.2008.172","article-title":"The Word Tree, an interactive visual concordance","volume":"14","author":"Wattenberg","year":"2008","journal-title":"IEEE Transact Visualization Comput Graphics."},{"issue":"2","key":"2020110612460725900_ocx070-B14","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1057\/palgrave.ivs.9500180","article-title":"Jigsaw: Supporting investigative analysis through interactive visualization","volume":"7","author":"Stasko","year":"2008","journal-title":"Inform Visualization."},{"issue":"5","key":"2020110612460725900_ocx070-B15","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1136\/amiajnl-2011-000183","article-title":"Automated concept-level information extraction to reduce the need for custom software and rules development","volume":"18","author":"D\u2019Avolio","year":"2011","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612460725900_ocx070-B16","first-page":"273","article-title":"Knowtator: a prot\u00e9g\u00e9 plug-in for annotated corpus construction","volume-title":"Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology","author":"Ogren"},{"issue":"5","key":"2020110612460725900_ocx070-B17","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","author":"Savova","year":"2010","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"2020110612460725900_ocx070-B18","doi-asserted-by":"crossref","first-page":"833","DOI":"10.1136\/amiajnl-2013-002255","article-title":"Assisted annotation of medical free text using RapTAT","volume":"21","author":"Gobbel","year":"2014","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612460725900_ocx070-B19","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1145\/2678025.2701399","article-title":"Principles of explanatory debugging to personalize interactive machine learning","volume-title":"Proceedings of the 20th International Conference on Intelligent User Interfaces","author":"Kulesza","year":"2015"},{"volume-title":"Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference","author":"Zaidan","article-title":"Using \u201cannotator rationales\u201d to improve machine learning for text categorization","key":"2020110612460725900_ocx070-B20"},{"author":"Trivedi","doi-asserted-by":"crossref","article-title":"An interactive tool for natural language processing on clinical text","key":"2020110612460725900_ocx070-B21","DOI":"10.1093\/jamia\/ocx070"},{"volume-title":"Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing","author":"Yessenalina","article-title":"Multi-level structured models for document-level sentiment classification","key":"2020110612460725900_ocx070-B22"},{"key":"2020110612460725900_ocx070-B23","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1136\/amiajnl-2011-000431","article-title":"Developing a natural language processing application for measuring the quality of colonoscopy procedures","volume":"18","author":"Harkema","year":"2011","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612460725900_ocx070-B24","article-title":"SUS: a quick and dirty usability scale","volume-title":"Usability Evaluation in Industry","author":"Brooke","year":"1996"},{"issue":"1","key":"2020110612460725900_ocx070-B25","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ Psychol Measurement."},{"issue":"2","key":"2020110612460725900_ocx070-B26","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","article-title":"Interactive machine learning for health informatics: when do we need the human-in-the-loop?","volume":"3","author":"Holzinger","year":"2016","journal-title":"Brain Inform."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/1\/81\/34149688\/ocx070.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/25\/1\/81\/34149688\/ocx070.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,29]],"date-time":"2022-07-29T12:40:38Z","timestamp":1659098438000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/25\/1\/81\/4004729"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,7,22]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,7,22]]},"published-print":{"date-parts":[[2018,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocx070","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"type":"print","value":"1067-5027"},{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2018,1]]},"published":{"date-parts":[[2017,7,22]]}}}