{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T12:33:25Z","timestamp":1761395605312},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"S1","license":[{"start":{"date-parts":[[2019,11,1]],"date-time":"2019-11-01T00:00:00Z","timestamp":1572566400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2019,11,12]],"date-time":"2019-11-12T00:00:00Z","timestamp":1573516800000},"content-version":"vor","delay-in-days":11,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Biomed Semant"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n<jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Significant amounts of health data are stored as free-text within clinical reports, letters, discharge summaries and notes. Busy clinicians have limited time to read such large amounts of free-text and are at risk of information overload and consequently missing information vital to patient care. Automatically identifying relevant information at the point of care has the potential to reduce these risks but represents a considerable research challenge. One software solution that has been proposed in industry is the IBM Watson analytics suite which includes rule-based analytics capable of processing large document collections at scale.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>In this paper we present an overview of IBM Watson Content Analytics and a feasibility study using Content Analytics with a large-scale corpus of clinical free-text reports within a UK National Health Service (NHS) context. We created dictionaries and rules for identifying positive incidence of hydronephrosis and brain metastasis from 5.6\u2009m radiology reports and were able to achieve 94% precision, 95% recall and 89% precision, 94% recall respectively on a sample of manually annotated reports. With minor changes for US English we applied the same rule set to an open access corpus of 0.5\u2009m radiology reports from a US hospital and achieved 93% precision, 94% recall and 84% precision, 88% recall respectively.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>We were able to implement IBM Watson within a UK NHS context and demonstrate effective results that could provide clinicians with an automatic safety net which highlights clinically important information within free-text documents. Our results suggest that currently available technologies such as IBM Watson Content Analytics already have the potential to address information overload and improve clinical safety and that solutions developed in one hospital and country may be transportable to different hospitals and countries. Our study was limited to exploring technical aspects of the feasibility of one industry solution and we recognise that healthcare text analytics research is a fast-moving field. That said, we believe our study suggests that text analytics is sufficiently advanced to be implemented within industry solutions that can improve clinical safety.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s13326-019-0213-5","type":"journal-article","created":{"date-parts":[[2019,11,12]],"date-time":"2019-11-12T01:02:36Z","timestamp":1573520556000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Finding relevant free-text radiology reports at scale with IBM Watson Content Analytics: a feasibility study in the UK NHS"],"prefix":"10.1186","volume":"10","author":[{"given":"Alicja","family":"Piotrkowicz","sequence":"first","affiliation":[]},{"given":"Owen","family":"Johnson","sequence":"additional","affiliation":[]},{"given":"Geoff","family":"Hall","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,11,12]]},"reference":[{"issue":"24","key":"213_CR1","first-page":"2479","volume":"311","author":"GM Weber","year":"2014","unstructured":"Weber GM, Mandl KD, Kohane IS. Finding the missing link for big biomedical data. JAMA. 2014;311(24):2479\u201380.","journal-title":"JAMA"},{"key":"213_CR2","doi-asserted-by":"publisher","first-page":"1079","DOI":"10.13063\/2327-9214.1079","volume":"2","author":"D Capurro","year":"2014","unstructured":"Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P. Availability of Structured and Unstructured Clinical Data for Comparative Effectiveness Research and Quality Improvement: A Multi-Site Assessment. EGEMS (Wash DC). 2014;2:1079. https:\/\/doi.org\/10.13063\/2327-9214.1079.","journal-title":"EGEMS (Wash DC)"},{"issue":"5","key":"213_CR3","doi-asserted-by":"publisher","first-page":"552","DOI":"10.1136\/amiajnl-2011-000203","volume":"18","author":"\u00d6 Uzuner","year":"2011","unstructured":"Uzuner \u00d6, South BR, Shen S, DuVall SL. 2010 i2b2\/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc. 2011;18(5):552\u20136.","journal-title":"J Am Med Inform Assoc"},{"key":"213_CR4","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1109\/SOSE.2013.46","volume-title":"Service Oriented System Engineering (SOSE), 2013 IEEE 7th International Symposium","author":"R Smith","year":"2013","unstructured":"Smith R, Xu J, Hima S, Johnson O. Gateway to the Cloud-Case Study: A Privacy-Aware Environment for Electronic Health Records Research. In: Service Oriented System Engineering (SOSE), 2013 IEEE 7th International Symposium; 2013. p. 292\u20137."},{"key":"213_CR5","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1109\/ICHI.2016.48","volume-title":"Healthcare Informatics (ICHI), 2016 IEEE International Conference","author":"S Crossfield","year":"2016","unstructured":"Crossfield S, Johnson O, Fleming T. Large Scale Infrastructure for Health Data Analytics. In: Healthcare Informatics (ICHI), 2016 IEEE International Conference; 2016. p. 306."},{"key":"213_CR6","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AE Johnson","year":"2016","unstructured":"Johnson AE, Pollard TJ, Shen L, Lehman LWH, Feng M, Ghassemi M, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3:160035. https:\/\/doi.org\/10.1038\/sdata.2016.35.","journal-title":"Sci Data"},{"key":"213_CR7","first-page":"391","volume":"2009","author":"HJ Lowe","year":"2009","unstructured":"Lowe HJ, Ferris TA, Hernandez PM, Weber SC. STRIDE \u2013 An Integrated Standards - Based Translational Research Informatics Platform. AMIA Annu Symp Proc. 2009;2009:391\u20135.","journal-title":"AMIA Annu Symp Proc"},{"issue":"7","key":"213_CR8","doi-asserted-by":"publisher","first-page":"515","DOI":"10.1016\/j.ijmedinf.2010.03.004","volume":"79","author":"Karthik Natarajan","year":"2010","unstructured":"Natarajan K, Stein D, Jain S, Elhadad N. An analysis of clinical queries in an electronic health record search utility. Int J Med Inform. 2010. https:\/\/doi.org\/10.1016\/j.ijmedinf.2010.03.004.","journal-title":"International Journal of Medical Informatics"},{"key":"213_CR9","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1016\/j.jbi.2015.05.003","volume":"55","author":"DA Hanauer","year":"2015","unstructured":"Hanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: a report of University of Michigan\u2019s nine-year experience in developing and using the electronic medical record search engine (EMERSE). J Biomed Inform. 2015;55:290\u2013300.","journal-title":"J Biomed Inform"},{"issue":"5","key":"213_CR10","doi-asserted-by":"publisher","first-page":"839","DOI":"10.1016\/j.jbi.2009.05.002","volume":"42","author":"Henk Harkema","year":"2009","unstructured":"Harkema H, Dowling JN, Thornblade T, Chapman WW. ConText: an algorithm for determining negation, experiencer, and temporal status from clinical reports. J Biomed Inform. 2009. https:\/\/doi.org\/10.1016\/j.jbi.2009.05.002.","journal-title":"Journal of Biomedical Informatics"},{"key":"213_CR11","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1016\/j.ijmedinf.2014.06.009","volume":"83","author":"I Spasi\u0107","year":"2014","unstructured":"Spasi\u0107 I, Livsey J, Keane JA, Nenadi\u0107 G. Text mining of cancer-related information: review of current status and future directions. Int J Med Inform. 2014;83:605\u201323. https:\/\/doi.org\/10.1016\/j.ijmedinf.2014.06.009.","journal-title":"Int J Med Inform"},{"key":"213_CR12","doi-asserted-by":"publisher","first-page":"760","DOI":"10.1016\/j.jbi.2009.08.007","volume":"42","author":"D Demner-Fushman","year":"2009","unstructured":"Demner-Fushman D, Chapman WW, McDonald CJ. What can natural language processing do for clinical decision support? J Biomed Inform. 2009;42:760\u201372. https:\/\/doi.org\/10.1016\/j.jbi.2009.08.007.","journal-title":"J Biomed Inform"},{"key":"213_CR13","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1136\/jamia.2009.002295","volume":"17","author":"RS Crowley","year":"2010","unstructured":"Crowley RS, Castine M, Mitchell K, Chavan G, McSherry T, Feldman M. caTIES: A grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research. J Am Med Informatics Assoc. 2010;17:253\u201364. https:\/\/doi.org\/10.1136\/jamia.2009.002295.","journal-title":"J Am Med Informatics Assoc"},{"key":"213_CR14","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J Am Med Informatics Assoc. 2010;17:507\u201313. https:\/\/doi.org\/10.1136\/jamia.2009.001560.","journal-title":"J Am Med Informatics Assoc"},{"issue":"4","key":"213_CR15","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1016\/j.clinthera.2015.12.001","volume":"38","author":"Y Chen","year":"2016","unstructured":"Chen Y, Argentinis JE, Weber G. IBM Watson: how cognitive computing can be applied to big data challenges in life sciences research. Clin Ther. 2016;38(4):688\u2013701.","journal-title":"Clin Ther"},{"key":"213_CR16","volume-title":"Reform Research Trust","author":"E Harwich","year":"2018","unstructured":"Harwich E, Laycock K. Thinking on its own: AI in the NHS. In: Reform Research Trust; 2018. Available online at www.reform.uk\/publication\/thinking-on-its-own-ai-in-the-nhs. Accessed 7 June 2018."},{"key":"213_CR17","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1188\/15.CJON.31-32","volume":"19","author":"S Doyle-Lindrud","year":"2012","unstructured":"Doyle-Lindrud S. Watson will see you now: a supercomputer to help clinicians make informed treatment decisions. Clin J Oncol Nurs. 2012;19:31\u20133. https:\/\/doi.org\/10.1188\/15.CJON.31-32.","journal-title":"Clin J Oncol Nurs"},{"key":"213_CR18","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1200\/JOP.2013.001021","volume":"9","author":"JL Malin","year":"2013","unstructured":"Malin JL. Envisioning Watson as a rapid-learning system for oncology. J Oncol Pract. 2013;9:155\u20137. https:\/\/doi.org\/10.1200\/JOP.2013.001021.","journal-title":"J Oncol Pract"},{"key":"213_CR19","doi-asserted-by":"publisher","unstructured":"Glass ZM, Gucalp A, Epstein AS, David SA, Caroline A, Granovsky S, et al. Piloting IBM Watson Oncology within Memorial Sloan Kettering\u2019s regional network. J Clin Oncol. 2014;32. https:\/\/doi.org\/10.1200\/jco.2014.32.15.","DOI":"10.1200\/jco.2014.32.15"},{"key":"213_CR20","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39146-0","volume-title":"Human-computer interaction and knowledge discovery in complex, unstructured, Big Data. HCI-KDD","author":"A Holzinger","year":"2013","unstructured":"Holzinger A, Ofner B, Hofmann-wellenhof R. Human-computer interaction and knowledge discovery in complex, unstructured, Big Data. HCI-KDD; 2013. https:\/\/doi.org\/10.1007\/978-3-642-39146-0."},{"key":"213_CR21","doi-asserted-by":"publisher","unstructured":"Stocker C, Marzi LM, Matula C, Schantl J, Prohaska G, Brabenetz A, et al. Enhancing patient safety through human-computer information retrieval on the example of german-speaking surgical reports. In: 2014 25th International Workshop on Database and Expert Systems Applications: DEXA; 2014. p. 216\u201320. https:\/\/doi.org\/10.1109\/DEXA.2014.53.","DOI":"10.1109\/DEXA.2014.53"},{"key":"213_CR22","unstructured":"Digital Health. Using IBM Watson cognitive technology to enhance the patient experience: Digital Health Net; 2018. Available online at www.digitalhealth.net\/2018\/03\/using-ibm-watson-cognitive-technology-to-enhance-the-patient-experience. Accessed 7 June 2018"},{"key":"213_CR23","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1017\/S1351324904003523","volume":"10","author":"D Ferrucci","year":"2004","unstructured":"Ferrucci D, Lally A. UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat Lang Eng. 2004;10:327\u201348. https:\/\/doi.org\/10.1017\/S1351324904003523.","journal-title":"Nat Lang Eng"},{"key":"213_CR24","volume-title":"UK Acad Inf Syst Conf Proc","author":"OA Johnson","year":"2011","unstructured":"Johnson OA, Abiodun SE. Understanding what success in health information systems looks like: the patient Pathway Management (PPM) system at Leeds. In: UK Acad Inf Syst Conf Proc; 2011."},{"key":"213_CR25","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1148\/radiol.16142770","volume":"279","author":"E Pons","year":"2016","unstructured":"Pons E, Braun LMM, Hunink MGM, Kors JA. Natural language processing in radiology: a systematic review. Radiology. 2016;279:329\u201343.","journal-title":"Radiology"},{"issue":"2","key":"213_CR26","doi-asserted-by":"publisher","first-page":"77","DOI":"10.14236\/jhi.v25i2.996","volume":"25","author":"S McLachlan","year":"2018","unstructured":"McLachlan S, Potts HWW, Dube K, Buchanan D, Lean S, Gallagher T, Johnson O, Daley B, Marsh W, Fenton N. The Heimdall Framework for supporting characterisation of Learning Health Systems. J Innov Health Inform. 2018;25(2):77\u201387.","journal-title":"J Innov Health Inform"}],"container-title":["Journal of Biomedical Semantics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-019-0213-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s13326-019-0213-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13326-019-0213-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,11]],"date-time":"2020-11-11T00:36:56Z","timestamp":1605055016000},"score":1,"resource":{"primary":{"URL":"https:\/\/jbiomedsem.biomedcentral.com\/articles\/10.1186\/s13326-019-0213-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":26,"journal-issue":{"issue":"S1","published-print":{"date-parts":[[2019,11]]}},"alternative-id":["213"],"URL":"https:\/\/doi.org\/10.1186\/s13326-019-0213-5","relation":{},"ISSN":["2041-1480"],"issn-type":[{"value":"2041-1480","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]},"assertion":[{"value":"12 November 2019","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The work was conducted by research and informatics staff contracted to Leeds Teaching Hospitals Trust Informatics Department using anonymised retrospective data from the Trust. Approval to perform the study was obtained from Leeds Teaching Hospitals Trust Research and Innovation Director and the Trust\u2019s Information Governance lead.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"21"}}