{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T15:01:53Z","timestamp":1771340513752,"version":"3.50.1"},"reference-count":70,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2019,7,15]],"date-time":"2019-07-15T00:00:00Z","timestamp":1563148800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Virtual patient software allows health professionals to practise their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history-taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we designed a patient record model, a knowledge model for the task and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowledge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 terms, and dictionaries with over 959,000 word\/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non-medical evaluators) interact with the system and use 35 cases from 18 specialities. We conducted a quantitative evaluation of all components by analysing interaction logs (11,834 turns). Natural language understanding achieved an F-measure of 95.8%. Dialogue management provided on average 74.3 (\u00b19.5)% of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All evaluated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8% of their terms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.<\/jats:p>","DOI":"10.1017\/s1351324919000329","type":"journal-article","created":{"date-parts":[[2019,7,15]],"date-time":"2019-07-15T07:49:40Z","timestamp":1563176980000},"page":"183-220","source":"Crossref","is-referenced-by-count":28,"title":["Designing a virtual patient dialogue system based on terminology-rich resources: Challenges and evaluation"],"prefix":"10.1017","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-1756","authenticated-orcid":false,"given":"Leonardo","family":"Campillos-Llanos","sequence":"first","affiliation":[]},{"given":"Catherine","family":"Thomas","sequence":"additional","affiliation":[]},{"given":"\u00c9ric","family":"Bilinski","sequence":"additional","affiliation":[]},{"given":"Pierre","family":"Zweigenbaum","sequence":"additional","affiliation":[]},{"given":"Sophie","family":"Rosset","sequence":"additional","affiliation":[]}],"member":"56","published-online":{"date-parts":[[2019,7,15]]},"reference":[{"key":"S1351324919000329_ref31","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-85483-8_40"},{"key":"S1351324919000329_ref27","first-page":"133","article-title":"The virtual standardized patient","volume":"70","author":"Hubal","year":"2000","journal-title":"Studies in Health Technology and Informatics"},{"key":"S1351324919000329_ref26","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2016.04.003"},{"key":"S1351324919000329_ref4","unstructured":"Bickmore, T. 2015. Conversational agents for automated inpatient and outpatient health counseling. In Proc. of the AMIA Symposium, San Francisco, USA, p. 2131."},{"key":"S1351324919000329_ref18","unstructured":"Dickerson, R. , Johnsen, K. , Raij, A. , Lok, B. , Hernandez, J. , Stevens, A. and Lind, D.S. 2005. Evaluating a script-based approach for simulating patient-doctor interaction. In Proc. of the Intern. Conference of Human-Computer Interface Advances for Modeling and Simulation, pp. 79\u201384."},{"key":"S1351324919000329_ref17","first-page":"451","volume-title":"Proc. of the International Conference on Intelligent Virtual Agents 2016","author":"Datta","year":"2016"},{"key":"S1351324919000329_ref1","volume-title":"Guide de l\u2019examen clinique \u2013 Nouvelle \u00e9dition 2014","author":"Bates","year":"2014"},{"key":"S1351324919000329_ref14","volume-title":"Guide de conversation m\u00e9dicale - fran\u00e7ais-anglais-allemand","author":"Coud\u00e9","year":"2011"},{"key":"S1351324919000329_ref10","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-2343"},{"key":"S1351324919000329_ref8","first-page":"438","volume-title":"Proc. of SIGDIAL","author":"Campillos-Llanos","year":"2015"},{"key":"S1351324919000329_ref70","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2004.03.010"},{"key":"S1351324919000329_ref6","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkh061"},{"key":"S1351324919000329_ref11","first-page":"49","volume-title":"Deep Learning in Natural Language Processing","author":"Celikyilmaz","year":"2017"},{"key":"S1351324919000329_ref39","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2004.11.006"},{"key":"S1351324919000329_ref49","volume-title":"Apprentissage par renforcement pour la g\u00e9n\u00e9ralisation des approches automatiques dans la conception des systemes de dialogue oral","author":"Pinault","year":"2011"},{"key":"S1351324919000329_ref37","unstructured":"McCray, A.T. , Srinivasan, S. and Browne, A.C. 1994. Lexical methods for managing variation in biomedical terminologies. In Proc. of Annual Symposium Computer Applic. Medical Care, Washington, pp. 235\u2013239."},{"key":"S1351324919000329_ref56","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-85760-0_55"},{"key":"S1351324919000329_ref34","first-page":"707","article-title":"Binary codes capable of correcting deletions, insertions, and reversals","volume":"10","author":"Levenshtein","year":"1966","journal-title":"Soviet Physics Doklady"},{"key":"S1351324919000329_ref51","volume-title":"A comprehensive grammar of the English language","author":"Quirk","year":"1985"},{"key":"S1351324919000329_ref3","doi-asserted-by":"publisher","DOI":"10.5688\/aj7408143"},{"key":"S1351324919000329_ref28","first-page":"127","article-title":"Lessons learned in modeling virtual pediatric patients","volume":"94","author":"Hubal","year":"2003","journal-title":"Studies in Health Technology and Informatics"},{"key":"S1351324919000329_ref19","first-page":"279","article-title":"SNOMED-CT: The advanced terminology and coding system for eHealth","volume":"121","author":"Donnelly","year":"2006","journal-title":"Studies in Health Technology and Informatics"},{"key":"S1351324919000329_ref13","doi-asserted-by":"publisher","DOI":"10.1097\/ACM.0b013e3181edfe13"},{"key":"S1351324919000329_ref21","volume-title":"Guide pratique de l\u2019examen clinique","author":"Epstein","year":"2015"},{"key":"S1351324919000329_ref35","doi-asserted-by":"publisher","DOI":"10.1097\/SIH.0000000000000195"},{"key":"S1351324919000329_ref63","first-page":"3","article-title":"Designing useful virtual standardized patient encounters","volume":"4","author":"Talbot","year":"2012b","journal-title":"Proc. of the Interservice\/Industry Training, Simulation and Education Conference"},{"key":"S1351324919000329_ref29","unstructured":"Jin, L. , White, M. , Jaffe, E. , Zimmerman, L. and Danforth, D. 2017. Combining CNNs and Pattern Matching for Question Interpretation in a Virtual Patient Dialogue System. In Proc. of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, Copenhagen, Denmark, pp. 11\u201321."},{"key":"S1351324919000329_ref45","unstructured":"Norvig, P. 2007. How to write a spelling corrector. http:\/\/norvig.com\/spell-correct.html. Accessed 23 April 2018."},{"key":"S1351324919000329_ref22","volume-title":"Approaches and methodologies for automatic Question-Answering in an open-domain, interactive setup","author":"Galibert","year":"2009"},{"key":"S1351324919000329_ref32","doi-asserted-by":"publisher","DOI":"10.4018\/978-1-60960-617-6.ch011"},{"key":"S1351324919000329_ref30","doi-asserted-by":"publisher","DOI":"10.2200\/S00204ED1V01Y200910HLT005"},{"key":"S1351324919000329_ref67","first-page":"126","volume-title":"Proc. of Interspeech","author":"van Schooten","year":"2007"},{"key":"S1351324919000329_ref12","first-page":"277","article-title":"Tools for research and education in speech science","volume":"1","author":"Cole","year":"1999","journal-title":"Proc. of the International Conference of Phonetic Sciences"},{"key":"S1351324919000329_ref44","unstructured":"Nirenburg, S. , McShane, M. and Beale, S. 2009. A unified ontological-semantic substrate for physiological simulation and cognitive modeling. In Proc. of International Conference on Biomedical Ontology (ICBO), Buffalo, New York, pp. 139\u2013142."},{"key":"S1351324919000329_ref24","first-page":"3174","volume-title":"Proc. of LREC 2016","author":"Gokcen","year":"2016"},{"key":"S1351324919000329_ref38","first-page":"216","article-title":"Aggregating UMLS semantic types for reducing conceptual complexity","volume":"84","author":"McCray","year":"2001","journal-title":"Studies in Health Technology and Informatics"},{"key":"S1351324919000329_ref25","first-page":"178","volume-title":"Many Morphologies","author":"Hathout","year":"2002"},{"key":"S1351324919000329_ref40","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2001.0080080"},{"key":"S1351324919000329_ref42","unstructured":"Nirenburg, S. , Beale, S. , McShane, M. , Jarrell, B. and Fantry, G. 2008. Language understanding in Maryland virtual patient. In Proc. of the 22nd International Conference on Computational Linguistics, pp. 36\u201339."},{"key":"S1351324919000329_ref33","first-page":"L3","volume-title":"Actes du colloque Serious Games en M\u00e9decine et Sant\u00e9 (SeGaMED) 2013","author":"Lelardeux","year":"2013"},{"key":"S1351324919000329_ref43","unstructured":"Nirenburg, S. , McShane, M. , Beale, S. and Jarrell, B. 2008. Adaptivity in a multi-agent clinical simulation system. In Proc. of AKRR\u201908, International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning, pp. 17\u201319."},{"key":"S1351324919000329_ref46","unstructured":"Paek, T. 2001. Empirical methods for evaluating dialog systems. In Proc. of the workshop on Evaluation for Language and Dialogue Systems-Volume 9, Toulouse, France, pp. 1\u20139."},{"key":"S1351324919000329_ref16","doi-asserted-by":"publisher","DOI":"10.4101\/jvwr.v2i2.707"},{"key":"S1351324919000329_ref47","volume-title":"How can I help you today? Guide de la consultation m\u00e9dicale et param\u00e9dicale en anglais","author":"Pastore","year":"2015"},{"key":"S1351324919000329_ref7","unstructured":"Bouamor, D. , Campillos-Llanos, L. , Ligozat, A.-L. , Rosset, S. and Zweigenbaum, P. 2016. Transfer-based learning-to-rank assessment of medical term technicality. In Calzolari, N. et al. (eds.), Proc. of LREC 2016, Portoro\u017e, Slovenia, pp. 2312\u20132316."},{"key":"S1351324919000329_ref5","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2005.12.004"},{"key":"S1351324919000329_ref48","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2011.11.008"},{"key":"S1351324919000329_ref50","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-010-0019-2_11"},{"key":"S1351324919000329_ref52","volume-title":"Patients virtuels: p\u00e9dagogie, \u00e9tat de l\u2019art et d\u00e9veloppement du simulateur Alphadiag","author":"Rombauts","year":"2014"},{"key":"S1351324919000329_ref41","unstructured":"Namer, F. and Zweigenbaum, P. 2004. Acquiring meaning for French medical terminology: contribution of morphosemantics. In Proc. of the 11th MEDINFO Conference, San Francisco, USA, pp. 535\u2013539."},{"key":"S1351324919000329_ref53","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04380-2_52"},{"key":"S1351324919000329_ref54","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2011.11.004"},{"key":"S1351324919000329_ref55","doi-asserted-by":"crossref","unstructured":"Rosset, S. , Galibert, O. , Illouz, G. and Max, A. Integrating Spoken Dialog and Question Answering: the Ritel Project Proc. of InterSpeech 2006, Pittsburgh, USA, pp. 1914\u20131917.","DOI":"10.21437\/Interspeech.2006-526"},{"key":"S1351324919000329_ref57","unstructured":"Roy, B. and Graham, T.N. 2008. Methods for evaluating software architecture: A survey. Technical Report 545, School of Computing, Queen\u2019s University at Kingston, Ontario, Canada."},{"key":"S1351324919000329_ref58","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2012.01.085"},{"key":"S1351324919000329_ref59","first-page":"79","volume-title":"Actes du colloque Serious Games en M\u00e9decine et Sant\u00e9 (SeGaMED) 2013","author":"Siregard","year":"2013"},{"key":"S1351324919000329_ref68","first-page":"271","volume-title":"Proc. of the 8th Conference of the European chapter of the Association for Computational Linguistics","author":"Walker","year":"1997"},{"key":"S1351324919000329_ref61","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"1998"},{"key":"S1351324919000329_ref9","first-page":"3167","volume-title":"Proc. of LREC 2016","author":"Campillos-Llanos","year":"2016"},{"key":"S1351324919000329_ref62","doi-asserted-by":"publisher","DOI":"10.4018\/jgcms.2012070101"},{"key":"S1351324919000329_ref36","unstructured":"Makhoul, J. , Kubala, F. , Schwartz, R. and Weischedel, R. 1999. Performance measures for information extraction. In Proc. of DARPA Broadcast News Workshop, Virginia, USA, pp. 249\u2013252."},{"key":"S1351324919000329_ref64","first-page":"407","article-title":"Natural language understanding performance and use considerations in virtual medical encounters","volume":"220","author":"Talbot","year":"2016","journal-title":"Studies in Health Technology and Informatics"},{"key":"S1351324919000329_ref65","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-010-0019-2_15"},{"key":"S1351324919000329_ref66","first-page":"1699","volume-title":"Proc. of LREC 2004","author":"Traum","year":"2004"},{"key":"S1351324919000329_ref23","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2004.04.026"},{"key":"S1351324919000329_ref2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2005.12.008"},{"key":"S1351324919000329_ref69","first-page":"8","volume-title":"Proc. of Spoken Language Technology Workshop","author":"Young","year":"2006"},{"key":"S1351324919000329_ref15","doi-asserted-by":"publisher","DOI":"10.3406\/lfr.1990.6323"},{"key":"S1351324919000329_ref60","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjsurg.2006.03.002"},{"key":"S1351324919000329_ref20","unstructured":"Ellaway, R. , Candler, C. , Greene, P. and Smothers, V. 2006. An architectural model for MedBiquitous virtual patients. http:\/\/groups.medbiq.org\/medbiq\/display\/VPWG\/MedBiquitous+Virtual+Patient+Architecture. Accessed 23 April 2018."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324919000329","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,23]],"date-time":"2022-09-23T22:13:54Z","timestamp":1663971234000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324919000329\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,15]]},"references-count":70,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["S1351324919000329"],"URL":"https:\/\/doi.org\/10.1017\/s1351324919000329","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,15]]}}}