{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T13:32:15Z","timestamp":1773840735064,"version":"3.50.1"},"reference-count":94,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T00:00:00Z","timestamp":1700524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T00:00:00Z","timestamp":1700524800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100011033","name":"Agencia Estatal de Investigaci\u00f3n","doi-asserted-by":"publisher","award":["ID2019-107194GB- I00\/AEI\/10.13039\/501100011033"],"award-info":[{"award-number":["ID2019-107194GB- I00\/AEI\/10.13039\/501100011033"]}],"id":[{"id":"10.13039\/501100011033","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010801","name":"Xunta de Galicia","doi-asserted-by":"publisher","award":["ED431C 2022\/44"],"award-info":[{"award-number":["ED431C 2022\/44"]}],"id":[{"id":"10.13039\/501100010801","id-type":"DOI","asserted-by":"publisher"}]},{"name":"CITIC","award":["ED431G 2019\/01"],"award-info":[{"award-number":["ED431G 2019\/01"]}]},{"DOI":"10.13039\/501100014597","name":"Universidade da Coru\u00f1a","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100014597","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2024,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Any machine learning (ML) model is highly dependent on the data it uses for learning, and this is even more important in the case of deep learning models. The problem is a data bottleneck, i.e. the difficulty in obtaining an adequate number of cases and quality data. Another issue is improving the learning process, which can be done by actively introducing experts into the learning loop, in what is known as human-in-the-loop (HITL) ML. We describe an ML model based on a neural network in which HITL techniques were used to resolve the data bottleneck problem for the treatment of pancreatic cancer. We first augmented the dataset using synthetic cases created by a generative adversarial network. We then launched an active learning (AL) process involving human experts as oracles to label both new cases and cases by the network found to be suspect. This AL process was carried out simultaneously with an interactive ML process in which feedback was obtained from humans in order to develop better synthetic cases for each iteration of training. We discuss the challenges involved in including humans in the learning process, especially in relation to human\u2013computer interaction, which is acquiring great importance in building ML models and can condition the success of a HITL approach. This paper also discusses the methodological approach adopted to address these challenges.<\/jats:p>","DOI":"10.1007\/s00521-023-09197-2","type":"journal-article","created":{"date-parts":[[2023,11,21]],"date-time":"2023-11-21T08:06:45Z","timestamp":1700554005000},"page":"2597-2616","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Addressing the data bottleneck in medical deep learning models using a human-in-the-loop machine learning approach"],"prefix":"10.1007","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4894-1067","authenticated-orcid":false,"given":"Eduardo","family":"Mosqueira-Rey","sequence":"first","affiliation":[]},{"given":"Elena","family":"Hern\u00e1ndez-Pereira","sequence":"additional","affiliation":[]},{"given":"Jos\u00e9","family":"Bobes-Bascar\u00e1n","sequence":"additional","affiliation":[]},{"given":"David","family":"Alonso-R\u00edos","sequence":"additional","affiliation":[]},{"given":"Alberto","family":"P\u00e9rez-S\u00e1nchez","sequence":"additional","affiliation":[]},{"given":"\u00c1ngel","family":"Fern\u00e1ndez-Leal","sequence":"additional","affiliation":[]},{"given":"Vicente","family":"Moret-Bonillo","sequence":"additional","affiliation":[]},{"given":"Yolanda","family":"Vidal-\u00cdnsua","sequence":"additional","affiliation":[]},{"given":"Francisca","family":"V\u00e1zquez-Rivera","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,11,21]]},"reference":[{"issue":"101","key":"9197_CR1","doi-asserted-by":"publisher","first-page":"813","DOI":"10.1016\/j.inffus.2023.101813","volume":"97","author":"M Abdar","year":"2023","unstructured":"Abdar M, Mehrzadi A, Goudarzi M et al (2023) Binarized multi-gate mixture of Bayesian experts for cardiac syndrome x diagnosis: a clinician-in-the-loop scenario with a belief-uncertainty fusion paradigm. Inf Fusion 97(101):813. https:\/\/doi.org\/10.1016\/j.inffus.2023.101813","journal-title":"Inf Fusion"},{"key":"9197_CR2","volume-title":"Handbook for evaluating knowledge-based systems: conceptual framework and compendium of methods","author":"L Adelman","year":"2012","unstructured":"Adelman L, Riedel SL (2012) Handbook for evaluating knowledge-based systems: conceptual framework and compendium of methods. Springer, Berlin"},{"key":"9197_CR3","doi-asserted-by":"publisher","unstructured":"Alonso-R\u00edos D, Raneburger D, Popp R et\u00a0al (2014) A user study on tailoring GUIs for smartphones. In: Proceedings of the 29th annual ACM symposium on applied computing. Association for Computing Machinery, New York, NY, USA, SAC \u201914, pp 186\u2013192. https:\/\/doi.org\/10.1145\/2554850.2555085","DOI":"10.1145\/2554850.2555085"},{"issue":"1","key":"9197_CR4","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1080\/10447310903025552","volume":"26","author":"D Alonso-R\u00edos","year":"2009","unstructured":"Alonso-R\u00edos D, V\u00e1zquez-Garc\u00eda A, Mosqueira-Rey E et al (2009) Usability: a critical analysis and a taxonomy. Int J Hum Comput Interact 26(1):53\u201374. https:\/\/doi.org\/10.1080\/10447310903025552","journal-title":"Int J Hum Comput Interact"},{"issue":"10","key":"9197_CR5","doi-asserted-by":"publisher","first-page":"941","DOI":"10.1080\/10447318.2010.502099","volume":"26","author":"D Alonso-R\u00edos","year":"2010","unstructured":"Alonso-R\u00edos D, V\u00e1zquez-Garc\u00eda A, Mosqueira-Rey E et al (2010) A context-of-use taxonomy for usability studies. Int J Hum Comput Interact 26(10):941\u2013970. https:\/\/doi.org\/10.1080\/10447318.2010.502099","journal-title":"Int J Hum Comput Interact"},{"issue":"12","key":"9197_CR6","doi-asserted-by":"publisher","first-page":"1169","DOI":"10.1080\/10447318.2018.1424101","volume":"34","author":"D Alonso-R\u00edos","year":"2018","unstructured":"Alonso-R\u00edos D, Mosqueira-Rey E, Moret-Bonillo V (2018) A systematic and generalizable approach to the heuristic evaluation of user interfaces. Int J Hum Comput Interact 34(12):1169\u20131182. https:\/\/doi.org\/10.1080\/10447318.2018.1424101","journal-title":"Int J Hum Comput Interact"},{"issue":"4","key":"9197_CR7","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1609\/aimag.v35i4.2513","volume":"35","author":"S Amershi","year":"2014","unstructured":"Amershi S, Cakmak M, Knox WB et al (2014) Power to the people: the role of humans in interactive machine learning. AI Mag 35(4):105\u2013120. https:\/\/doi.org\/10.1609\/aimag.v35i4.2513","journal-title":"AI Mag"},{"issue":"2","key":"9197_CR8","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1145\/3517337","volume":"29","author":"L Aroyo","year":"2022","unstructured":"Aroyo L, Lease M, Paritosh P, Schaekermann M (2022) Data excellence for AI: Why should you care? Interactions 29(2):66\u201369. https:\/\/doi.org\/10.1145\/3517337","journal-title":"Interactions"},{"issue":"9","key":"9197_CR9","doi-asserted-by":"publisher","first-page":"875","DOI":"10.1080\/1206212X.2021.1974663","volume":"44","author":"DM Belete","year":"2022","unstructured":"Belete DM, Huchaiah MD (2022) Grid search in hyperparameter optimization of machine learning models for prediction of HIV\/AIDS test results. Int J Comput Appl 44(9):875\u2013886. https:\/\/doi.org\/10.1080\/1206212X.2021.1974663","journal-title":"Int J Comput Appl"},{"key":"9197_CR10","doi-asserted-by":"publisher","unstructured":"Bengio Y, Louradour J, Collobert R et\u00a0al (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning. Association for Computing Machinery, New York, NY, USA, ICML \u201909, pp 41\u201348. https:\/\/doi.org\/10.1145\/1553374.1553380","DOI":"10.1145\/1553374.1553380"},{"key":"9197_CR11","doi-asserted-by":"crossref","unstructured":"Blackwell AF, Britton C, Cox AL et\u00a0al (2001) Cognitive dimensions of notations: design tools for cognitive technology. In: Proceedings of the 4th international conference on cognitive technology: instruments of mind. Springer-Verlag, Berlin, Heidelberg, CT \u201901, pp 325\u2013341","DOI":"10.1007\/3-540-44617-6_31"},{"key":"9197_CR12","unstructured":"Boecking B, Neiswanger W, Xing E et\u00a0al (2021) Interactive weak supervision: Learning useful heuristics for data labeling. arXiv:2012.06046"},{"key":"9197_CR13","doi-asserted-by":"publisher","unstructured":"Bravo-Rocca G, Liu P, Guitart J et\u00a0al (2022) Human-in-the-loop online multi-agent approach to increase trustworthiness in ml models through trust scores and data augmentation. In: 2022 IEEE 46th annual computers, software, and applications conference (COMPSAC), pp 32\u201337. https:\/\/doi.org\/10.1109\/COMPSAC54236.2022.00014","DOI":"10.1109\/COMPSAC54236.2022.00014"},{"issue":"1","key":"9197_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s42486-022-00115-4","volume":"5","author":"L Chen","year":"2023","unstructured":"Chen L, Wang J, Guo B et al (2023) Human-in-the-loop machine learning with applications for population health. CCF Trans Pervasive Comput Interact 5(1):1\u201312. https:\/\/doi.org\/10.1007\/s42486-022-00115-4","journal-title":"CCF Trans Pervasive Comput Interact"},{"key":"9197_CR15","unstructured":"Choi E, Biswal S, Malin B et\u00a0al (2017) Generating multi-label discrete patient records using generative adversarial networks. In: Doshi-Velez F, Fackler J, Kale D et\u00a0al (eds) Proceedings of the 2nd machine learning for healthcare conference, proceedings of machine learning research, vol\u00a068. PMLR, pp 286\u2013305. https:\/\/proceedings.mlr.press\/v68\/choi17a.html"},{"issue":"120","key":"9197_CR16","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1016\/j.eswa.2023.120216","volume":"226","author":"R Delussu","year":"2023","unstructured":"Delussu R, Putzu L, Fumera G (2023) Human-in-the-loop cross-domain person re-identification. Expert Syst Appl 226(120):216. https:\/\/doi.org\/10.1016\/j.eswa.2023.120216","journal-title":"Expert Syst Appl"},{"key":"9197_CR17","doi-asserted-by":"publisher","unstructured":"Donmez P, Carbonell JG (2008) Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In: Proceedings of the 17th ACM conference on information and knowledge management. Association for Computing Machinery, New York, NY, USA, CIKM \u201908, pp 619\u2013628. https:\/\/doi.org\/10.1145\/1458082.1458165,","DOI":"10.1145\/1458082.1458165"},{"key":"9197_CR18","unstructured":"Fiebrink R, Cook PR (2010) The wekinator: a system for real-time, interactive machine learning in music. In: Proceedings of the eleventh international society for music information retrieval conference (ISMIR 2010), Utrecht"},{"key":"9197_CR19","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1016\/j.ins.2021.06.083","volume":"576","author":"J Fu","year":"2021","unstructured":"Fu J, Li W, Du J et al (2021) Dsagan: a generative adversarial network based on dual-stream attention mechanism for anatomical and functional image fusion. Inf Sci 576:484\u2013506. https:\/\/doi.org\/10.1016\/j.ins.2021.06.083","journal-title":"Inf Sci"},{"key":"9197_CR20","volume-title":"Advances in neural information processing systems","author":"I Goodfellow","year":"2014","unstructured":"Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C et al (eds) Advances in neural information processing systems, vol 27. Curran Associates Inc., New York"},{"issue":"3","key":"9197_CR21","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1207\/s15327051hci1303_2","volume":"13","author":"WD Gray","year":"1998","unstructured":"Gray WD, Salzman MC (1998) Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum Comput Interact 13(3):203\u2013261","journal-title":"Hum Comput Interact"},{"key":"9197_CR22","unstructured":"Gunning D (2017) Explainable artificial intelligence (xAI). Tech. rep., Defense Advanced Research Projects Agency (DARPA). https:\/\/www.darpa.mil\/program\/explainable-artificial-intelligence"},{"issue":"4","key":"9197_CR23","doi-asserted-by":"publisher","first-page":"5889","DOI":"10.1007\/s11042-021-11822-y","volume":"81","author":"K Guo","year":"2022","unstructured":"Guo K, Hu X, Li X (2022) Mmfgan: a novel multimodal brain medical image fusion based on the improvement of generative adversarial network. Multimedia Tools Appl 81(4):5889\u20135927","journal-title":"Multimedia Tools Appl"},{"key":"9197_CR24","doi-asserted-by":"publisher","first-page":"437","DOI":"10.1007\/978-3-031-25069-9_29","volume-title":"Computer vision\u2013ECCV 2022 workshops","author":"A Gupta","year":"2023","unstructured":"Gupta A, Sintorn IM (2023) Towards better guided attention and human knowledge insertion in deep convolutional neural networks. In: Karlinsky L, Michaeli T, Nishino K (eds) Computer vision\u2013ECCV 2022 workshops. Springer, Cham, pp 437\u2013453"},{"issue":"103","key":"9197_CR25","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1016\/j.jbi.2019.103136","volume":"92","author":"A Halder","year":"2019","unstructured":"Halder A, Kumar A (2019) Active learning using rough fuzzy classifier for cancer prediction from microarray gene expression data. J Biomed Inform 92(103):136. https:\/\/doi.org\/10.1016\/j.jbi.2019.103136","journal-title":"J Biomed Inform"},{"issue":"1","key":"9197_CR26","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1016\/j.tics.2014.10.004","volume":"19","author":"TT Hills","year":"2015","unstructured":"Hills TT, Todd PM, Lazer D et al (2015) Exploration versus exploitation in space, mind, and society. Trends Cogn Sci 19(1):46\u201354. https:\/\/doi.org\/10.1016\/j.tics.2014.10.004","journal-title":"Trends Cogn Sci"},{"issue":"13","key":"9197_CR27","first-page":"1717","volume":"25","author":"A Hinderks","year":"2019","unstructured":"Hinderks A, Winter D, Schrepp M et al (2019) Applicability of user experience and usability questionnaires. J Univ Comput Sci 25(13):1717\u20131735","journal-title":"J Univ Comput Sci"},{"key":"9197_CR28","doi-asserted-by":"publisher","unstructured":"Holmberg L, Davidsson P, Linde P (2020) A feature space focus in machine teaching. In: 2020 IEEE international conference on pervasive computing and communications workshops (PerCom Workshops), pp 1\u20132. https:\/\/doi.org\/10.1109\/PerComWorkshops48775.2020.9156175. http:\/\/mau.diva-portal.org\/smash\/get\/diva2:1428195\/FULLTEXT01.pdf","DOI":"10.1109\/PerComWorkshops48775.2020.9156175"},{"issue":"2","key":"9197_CR29","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1007\/s40708-016-0042-6","volume":"3","author":"A Holzinger","year":"2016","unstructured":"Holzinger A (2016) Interactive machine learning for health informatics: When do we need the human-in-the-loop? Brain Inform 3(2):119\u2013131. https:\/\/doi.org\/10.1007\/s40708-016-0042-6","journal-title":"Brain Inform"},{"key":"9197_CR30","unstructured":"Holzinger A, Biemann C, Pattichis CS et\u00a0al (2017) What do we need to build explainable AI systems for the medical domain? arXiv e-prints arXiv:1712.09923 [cs.AI]"},{"issue":"7","key":"9197_CR31","doi-asserted-by":"publisher","first-page":"2401","DOI":"10.1007\/s10489-018-1361-5","volume":"49","author":"A Holzinger","year":"2019","unstructured":"Holzinger A, Plass M, Kickmeier-Rust M et al (2019) Interactive machine learning: experimental evidence for the human in the algorithmic loop. Appl Intell 49(7):2401\u20132414. https:\/\/doi.org\/10.1007\/s10489-018-1361-5","journal-title":"Appl Intell"},{"issue":"4","key":"9197_CR32","first-page":"228","volume":"19","author":"DS Hong","year":"2021","unstructured":"Hong DS, Baik C (2021) Generating and validating synthetic training data for predicting bankruptcy of individual businesses. J Inf Commun Converg Eng 19(4):228\u2013233","journal-title":"J Inf Commun Converg Eng"},{"key":"9197_CR33","unstructured":"ISO 9241-210:2010 (2010) ISO 9241-210:2010\u2014ergonomics of human\u2013system interaction\u2014Part 210: Human-centred design for interactive systems"},{"issue":"4","key":"9197_CR34","doi-asserted-by":"publisher","first-page":"470","DOI":"10.1145\/503112.503114","volume":"33","author":"MY Ivory","year":"2001","unstructured":"Ivory MY, Hearst MA (2001) The state of the art in automating usability evaluation of user interfaces. ACM Comput Surv 33(4):470\u2013516. https:\/\/doi.org\/10.1145\/503112.503114","journal-title":"ACM Comput Surv"},{"key":"9197_CR35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10278-021-00556-w","volume":"35","author":"JJ Jeong","year":"2022","unstructured":"Jeong JJ, Tariq A, Adejumo T et al (2022) Systematic review of generative adversarial networks (GANs) for medical image classification and segmentation. J Digit Imaging 35:1\u201316","journal-title":"J Digit Imaging"},{"key":"9197_CR36","doi-asserted-by":"publisher","unstructured":"Jia J, Wu P, Zhang K et\u00a0al (2022) Imbalanced disk failure data processing method based on CTGAN. In: Intelligent computing theories and application: 18th international conference, ICIC 2022, Xi\u2019an, China, August 7\u201311, 2022, Proceedings, Part II. Springer, Berlin, Heidelberg, pp 638\u2013649. https:\/\/doi.org\/10.1007\/978-3-031-13829-4_55","DOI":"10.1007\/978-3-031-13829-4_55"},{"key":"9197_CR37","doi-asserted-by":"crossref","unstructured":"Jiang J, Hu YC, Tyagi N et\u00a0al (2018) Tumor-aware, adversarial domain adaptation from CT to MRI for lung cancer segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 777\u2013785","DOI":"10.1007\/978-3-030-00934-2_86"},{"key":"9197_CR38","doi-asserted-by":"publisher","DOI":"10.1016\/j.compmedimag.2021.101969","author":"M Jiang","year":"2021","unstructured":"Jiang M, Zhi M, Wei L et al (2021) FA-GAN: fused attentive generative adversarial networks for MRI image super-resolution. Comput Med Imaging Graph. https:\/\/doi.org\/10.1016\/j.compmedimag.2021.101969","journal-title":"Comput Med Imaging Graph"},{"issue":"12","key":"9197_CR39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.isci.2021.103415","volume":"24","author":"J Ju","year":"2021","unstructured":"Ju J, Wismans LV, Mustafa DA et al (2021) Robust deep learning model for prognostic stratification of pancreatic ductal adenocarcinoma patients. Iscience 24(12):1\u201318","journal-title":"Iscience"},{"issue":"1","key":"9197_CR40","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1038\/nmeth.2281","volume":"10","author":"M Kabra","year":"2013","unstructured":"Kabra M, Robie AA, Rivera-Alba M et al (2013) Jaaba: interactive machine learning for automatic annotation of animal behavior. Nat Methods 10(1):64\u201367. https:\/\/doi.org\/10.1038\/nmeth.2281","journal-title":"Nat Methods"},{"key":"9197_CR41","doi-asserted-by":"publisher","first-page":"124,475","DOI":"10.1109\/ACCESS.2022.3218067","volume":"10","author":"W Khan","year":"2022","unstructured":"Khan W, Zaki N, Ahmad A et al (2022) Mixed data imputation using generative adversarial networks. IEEE Access 10:124,475-124,490. https:\/\/doi.org\/10.1109\/ACCESS.2022.3218067","journal-title":"IEEE Access"},{"key":"9197_CR42","doi-asserted-by":"publisher","unstructured":"Khanal S, Refati R, Glandt K et\u00a0al (2021) Using content analysis and machine learning to identify Covid-19 information relevant to low-income households on social media. In: 2021 IEEE international conference on parallel and distributed processing with applications, big data and cloud computing, sustainable computing and communications, social computing and networking (ISPA\/BDCloud\/SocialCom\/SustainCom), pp 1522\u20131531. https:\/\/doi.org\/10.1109\/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00205","DOI":"10.1109\/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00205"},{"issue":"3","key":"9197_CR43","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1111\/j.1467-8535.1993.tb00076.x","volume":"24","author":"J Kirakowski","year":"1993","unstructured":"Kirakowski J, Corbett M (1993) SUMI: the software usability measurement inventory. Br J Educ Technol 24(3):210\u2013212. https:\/\/doi.org\/10.1111\/j.1467-8535.1993.tb00076.x","journal-title":"Br J Educ Technol"},{"issue":"7553","key":"9197_CR44","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436\u2013444. https:\/\/doi.org\/10.1038\/nature14539","journal-title":"Nature"},{"key":"9197_CR45","doi-asserted-by":"publisher","unstructured":"Lee S, Amgad M, Masoud M et\u00a0al (2019) An ensemble-based active learning for breast cancer classification. In: 2019 IEEE international conference on bioinformatics and Biomedicine (BIBM), pp 2549\u20132553. https:\/\/doi.org\/10.1109\/BIBM47256.2019.8983317","DOI":"10.1109\/BIBM47256.2019.8983317"},{"issue":"9","key":"9197_CR46","doi-asserted-by":"publisher","first-page":"663","DOI":"10.1080\/10447318.2014.930311","volume":"30","author":"JR Lewis","year":"2014","unstructured":"Lewis JR (2014) Usability: lessons learned... and yet to be learned. Int J Hum Comput Interact 30(9):663\u2013684","journal-title":"Int J Hum Comput Interact"},{"key":"9197_CR47","doi-asserted-by":"crossref","unstructured":"Lison P, Hubin A, Barnes J et\u00a0al (2020) Named entity recognition without labelled data: a weak supervision approach. arXiv:2004.14723","DOI":"10.18653\/v1\/2020.acl-main.139"},{"issue":"2","key":"9197_CR48","first-page":"3","volume":"8","author":"AM Lund","year":"2001","unstructured":"Lund AM (2001) Measuring usability with the use questionnaire. Usab Interface 8(2):3\u20136","journal-title":"Usab Interface"},{"key":"9197_CR49","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2022.119350","author":"J Mendes","year":"2023","unstructured":"Mendes J, Pereira T, Silva F et al (2023) Lung CT image synthesis using GANS. Expert Syst Appl. https:\/\/doi.org\/10.1016\/j.eswa.2022.119350","journal-title":"Expert Syst Appl"},{"key":"9197_CR50","unstructured":"Meza\u00a0Mart\u00ednez MA, Nadj M, Maedche A (2019) Towards an integrative theoretical framework of interactive machine learning systems. In: Proceedings of the 27th European conference on information systems (ECIS), Stockholm, Uppsala, Sweden. https:\/\/aisel.aisnet.org\/ecis2019_rp\/172"},{"key":"9197_CR51","unstructured":"Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784"},{"key":"9197_CR52","doi-asserted-by":"publisher","first-page":"205,327","DOI":"10.1109\/ACCESS.2020.3037063","volume":"8","author":"J Moon","year":"2020","unstructured":"Moon J, Jung S, Park S et al (2020) Conditional tabular GAN-based two-stage data generation scheme for short-term load forecasting. IEEE Access 8:205,327-205,339. https:\/\/doi.org\/10.1109\/ACCESS.2020.3037063","journal-title":"IEEE Access"},{"key":"9197_CR53","unstructured":"Moreno Planas JM, S\u00e1nchez Ortega A, Garc\u00eda Bueno JM et al (2010) Avances en C\u00e1ncer de P\u00e1ncreas, del laboratorio a la cl\u00ednica. UCLM, AECC"},{"key":"9197_CR54","doi-asserted-by":"publisher","first-page":"553","DOI":"10.1016\/j.procs.2021.08.057","volume":"192","author":"E Mosqueira-Rey","year":"2021","unstructured":"Mosqueira-Rey E, Alonso-R\u00edos D, Baamonde-Lozano A (2021) Integrating iterative machine teaching and active learning into the machine learning loop. Procedia Comput Sci 192:553\u2013562. https:\/\/doi.org\/10.1016\/j.procs.2021.08.057","journal-title":"Procedia Comput Sci"},{"key":"9197_CR55","doi-asserted-by":"crossref","unstructured":"Mosqueira-Rey E, Fern\u00e1ndez-Casta\u00f1o S, Alonso-R\u00edos D et\u00a0al (2023a) Gamifying machine teaching: human-in-the-loop approach for diphthong and hiatus identification in Spanish language. In: knowledge-based and intelligent information and engineering systems: proceedings of the 27th international conference KES2023","DOI":"10.1016\/j.procs.2023.10.302"},{"key":"9197_CR56","doi-asserted-by":"publisher","first-page":"3005","DOI":"10.1007\/s10462-022-10246-w","volume":"56","author":"E Mosqueira-Rey","year":"2023","unstructured":"Mosqueira-Rey E, Hern\u00e1ndez-Pereira E, Alonso-R\u00edos D et al (2023) Human-in-the-loop machine learning: a state of the art. Artif Intell Rev 56:3005\u20133054. https:\/\/doi.org\/10.1007\/s10462-022-10246-w","journal-title":"Artif Intell Rev"},{"key":"9197_CR57","unstructured":"Mottini A, Lh\u00e9ritier A, Acuna-Agost R (2018) Airline passenger name record generation using generative adversarial networks. arXiv:1807.06657"},{"issue":"5","key":"9197_CR58","doi-asserted-by":"publisher","first-page":"897","DOI":"10.1038\/s41591-022-01804-4","volume":"28","author":"S Mullainathan","year":"2022","unstructured":"Mullainathan S, Obermeyer Z (2022) Solving medicine\u2019s data bottleneck: nightingale open science. Nat Med 28(5):897\u2013899. https:\/\/doi.org\/10.1038\/s41591-022-01804-4","journal-title":"Nat Med"},{"key":"9197_CR59","volume-title":"Human-in-the-loop machine learning","author":"R Munro","year":"2020","unstructured":"Munro R (2020) Human-in-the-loop machine learning. Manning Publications, New York"},{"issue":"119","key":"9197_CR60","doi-asserted-by":"publisher","first-page":"086","DOI":"10.1016\/j.actamat.2023.119086","volume":"255","author":"J Na","year":"2023","unstructured":"Na J, Kim SJ, Kim H et al (2023) A unified microstructure segmentation approach via human-in-the-loop machine learning. Acta Mater 255(119):086. https:\/\/doi.org\/10.1016\/j.actamat.2023.119086","journal-title":"Acta Mater"},{"key":"9197_CR61","unstructured":"NCCN (2019) Pancreatic adenocarcinoma, version 3.2019. National Comprehensive Cancer Network. http:\/\/pancreatic.altervista.org\/downloads\/NCCN3.2019Pancreatic.pdf"},{"issue":"1","key":"9197_CR62","doi-asserted-by":"publisher","first-page":"126","DOI":"10.3390\/cancers11010126","volume":"11","author":"R Nicolle","year":"2019","unstructured":"Nicolle R, Raffenne J, Paradis V et al (2019) Prognostic biomarkers in pancreatic cancer: avoiding errata when using the TCGA dataset. Cancers 11(1):126","journal-title":"Cancers"},{"key":"9197_CR63","unstructured":"Nielsen J (2020) 10 usability heuristics for user interface design. https:\/\/www.nngroup.com\/articles\/ten-usability-heuristics\/. Accessed: 01 May 2023"},{"key":"9197_CR64","doi-asserted-by":"crossref","unstructured":"Nielsen J, Molich R (1990) Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp 249\u2013256","DOI":"10.1145\/97243.97281"},{"key":"9197_CR65","doi-asserted-by":"publisher","DOI":"10.48129\/kjs.splml.19119","author":"RA Nugraha","year":"2022","unstructured":"Nugraha RA, Parede HF, Subekti A (2022) Oversampling based on generative adversarial networks to overcome imbalance data in predicting fraud insurance claim. Kuwait J Sci. https:\/\/doi.org\/10.48129\/kjs.splml.19119","journal-title":"Kuwait J Sci"},{"issue":"10","key":"9197_CR66","doi-asserted-by":"publisher","first-page":"1071","DOI":"10.14778\/3231751.3231757","volume":"11","author":"N Park","year":"2018","unstructured":"Park N, Mohammadi M, Gorde K et al (2018) Data synthesis based on generative adversarial networks. Proc VLDB Endow 11(10):1071\u20131083. https:\/\/doi.org\/10.14778\/3231751.3231757","journal-title":"Proc VLDB Endow"},{"key":"9197_CR67","doi-asserted-by":"publisher","unstructured":"Patki N, Wedge R, Veeramachaneni K (2016) The synthetic data vault. In: 2016 IEEE international conference on data science and advanced analytics (DSAA), pp 399\u2013410. https:\/\/doi.org\/10.1109\/DSAA.2016.49","DOI":"10.1109\/DSAA.2016.49"},{"issue":"5","key":"9197_CR68","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1109\/MCSE.2013.74","volume":"15","author":"R Porter","year":"2013","unstructured":"Porter R, Theiler J, Hush D (2013) Interactive machine learning in data exploitation. Comput Sci Eng 15(5):12\u201320. https:\/\/doi.org\/10.1109\/MCSE.2013.74","journal-title":"Comput Sci Eng"},{"key":"9197_CR69","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2022.3201153","author":"AU Rahman","year":"2022","unstructured":"Rahman AU, Al-Obeidat F, Tubaishat A et al (2022) Discovering the correlation between phishing susceptibility causing data biases and big five personality traits using C-GAN. IEEE Trans Comput Soc Syst. https:\/\/doi.org\/10.1109\/TCSS.2022.3201153","journal-title":"IEEE Trans Comput Soc Syst"},{"issue":"5\u20136","key":"9197_CR70","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1080\/07370024.2020.1734931","volume":"35","author":"G Ramos","year":"2020","unstructured":"Ramos G, Meek C, Simard P et al (2020) Interactive machine teaching: a human-centered approach to building machine-learned models. Hum Comput Interact 35(5\u20136):413\u2013451. https:\/\/doi.org\/10.1080\/07370024.2020.1734931","journal-title":"Hum Comput Interact"},{"key":"9197_CR71","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1007\/978-3-642-40480-1_34","volume-title":"Human\u2013computer interaction\u2014INTERACT 2013","author":"D Raneburger","year":"2013","unstructured":"Raneburger D, Alonso-R\u00edos D, Popp R et al (2013) A user study with GUIs tailored for smartphones. In: Kotz\u00e9 P, Marsden G, Lindgaard G et al (eds) Human\u2013computer interaction\u2014INTERACT 2013. Springer, Berlin, Heidelberg, pp 505\u2013512"},{"key":"9197_CR72","doi-asserted-by":"publisher","unstructured":"Raneburger D, Popp R, Alonso-R\u00edos D et\u00a0al (2013b) A user study with GUIs tailored for smartphones and tablet PCs. In: 2013 IEEE international conference on systems, man, and cybernetics, pp 3727\u20133732. https:\/\/doi.org\/10.1109\/SMC.2013.635","DOI":"10.1109\/SMC.2013.635"},{"key":"9197_CR73","doi-asserted-by":"publisher","first-page":"809","DOI":"10.1007\/978-1-4899-7637-6_24","volume-title":"Recommender systems handbook","author":"N Rubens","year":"2015","unstructured":"Rubens N, Elahi M, Sugiyama M et al (2015) Active learning in recommender systems. In: Ricci F, Rokach L, Shapira B (eds) Recommender systems handbook. Springer, Berlin, pp 809\u2013846. https:\/\/doi.org\/10.1007\/978-1-4899-7637-6_24"},{"key":"9197_CR74","volume-title":"Deep learning with structured data","author":"M Ryan","year":"2020","unstructured":"Ryan M (2020) Deep learning with structured data. Manning Publications Co, New York"},{"issue":"2","key":"9197_CR75","doi-asserted-by":"publisher","first-page":"1247","DOI":"10.1007\/s00521-022-07995-8","volume":"35","author":"F Saghir","year":"2023","unstructured":"Saghir F, Gonzalez Perdomo ME, Behrenbruch P (2023) Application of streaming analytics for artificial lift systems: a human-in-the-loop approach for analysing clustered time-series data from progressive cavity pumps. Neural Comput Appl 35(2):1247\u20131277. https:\/\/doi.org\/10.1007\/s00521-022-07995-8","journal-title":"Neural Comput Appl"},{"key":"9197_CR76","unstructured":"Settles B (2009) Active learning literature survey. Technical report, University of Wisconsin-Madison. Department of Computer Sciences. https:\/\/minds.wisconsin.edu\/handle\/1793\/60660"},{"issue":"2","key":"9197_CR77","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1145\/3551635","volume":"66","author":"C Shani","year":"2023","unstructured":"Shani C, Zarecki J, Shahaf D (2023) The lean data scientist: recent advances toward overcoming the data bottleneck. Commun ACM 66(2):92\u2013102. https:\/\/doi.org\/10.1145\/3551635","journal-title":"Commun ACM"},{"key":"9197_CR78","doi-asserted-by":"crossref","unstructured":"Shin HC, Tenenholtz NA, Rogers JK et\u00a0al (2018) Medical image synthesis for data augmentation and anonymization using generative adversarial networks. In: International workshop on simulation and synthesis in medical imaging. Springer, pp 1\u201311","DOI":"10.1007\/978-3-030-00536-8_1"},{"key":"9197_CR79","unstructured":"Simard PY, Amershi S, Chickering DM et\u00a0al (2017) Machine teaching: a new paradigm for building machine learning systems. arXiv e-prints arXiv:1707.06742"},{"issue":"1","key":"9197_CR80","doi-asserted-by":"publisher","first-page":"1212","DOI":"10.1038\/s41598-020-58290-2","volume":"10","author":"M Sinkala","year":"2020","unstructured":"Sinkala M, Mulder N, Martin D (2020) Machine learning and network analyses reveal disease subtypes of pancreatic cancer and their molecular characteristics. Sci Rep 10(1):1212","journal-title":"Sci Rep"},{"issue":"01","key":"9197_CR81","doi-asserted-by":"publisher","first-page":"482","DOI":"10.2118\/204224-PA","volume":"26","author":"J Tang","year":"2021","unstructured":"Tang J, Fan B, Xiao L et al (2021) A new ensemble machine-learning framework for searching sweet spots in shale reservoirs. SPE J 26(01):482\u2013497. https:\/\/doi.org\/10.2118\/204224-PA","journal-title":"SPE J"},{"key":"9197_CR82","doi-asserted-by":"publisher","unstructured":"Tlachac M, Gerych W, Agrawal K et\u00a0al (2022) Text generation to aid depression detection: a comparative study of conditional sequence generative adversarial networks. In: 2022 IEEE international conference on big data (big data), pp 2804\u20132813. https:\/\/doi.org\/10.1109\/BigData55660.2022.10020224,","DOI":"10.1109\/BigData55660.2022.10020224"},{"issue":"1A","key":"9197_CR83","doi-asserted-by":"publisher","first-page":"68","DOI":"10.5114\/wo.2014.47136","volume":"19","author":"K Tomczak","year":"2015","unstructured":"Tomczak K, Czerwi\u0144ska P, Wiznerowicz M (2015) The cancer genome atlas (TCGA): an immeasurable source of knowledge. Contemp Oncol 19(1A):68\u201377. https:\/\/doi.org\/10.5114\/wo.2014.47136","journal-title":"Contemp Oncol"},{"issue":"101","key":"9197_CR84","first-page":"308","volume":"72","author":"A Wali","year":"2022","unstructured":"Wali A, Alamgir Z, Karim S et al (2022) Generative adversarial networks for speech processing: a review. Comput Speech Lang 72(101):308","journal-title":"Comput Speech Lang"},{"key":"9197_CR85","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1016\/j.cose.2022.102956","volume":"124","author":"H Wang","year":"2023","unstructured":"Wang H, Miller DJ, Kesidis G (2023) Anomaly detection of adversarial examples using class-conditional generative adversarial networks. Comput Secur 124:10. https:\/\/doi.org\/10.1016\/j.cose.2022.102956","journal-title":"Comput Secur"},{"issue":"14","key":"9197_CR86","doi-asserted-by":"publisher","first-page":"10","DOI":"10.3390\/s22145243","volume":"22","author":"J Wang","year":"2022","unstructured":"Wang J, Yan X, Liu L et al (2022) Cttgan: traffic data synthesizing scheme based on conditional GAN. Sensors 22(14):10. https:\/\/doi.org\/10.3390\/s22145243","journal-title":"Sensors"},{"issue":"2","key":"9197_CR87","first-page":"1","volume":"54","author":"Z Wang","year":"2021","unstructured":"Wang Z, She Q, Ward TE (2021) Generative adversarial networks in computer vision: a survey and taxonomy. ACM Comput Surv (CSUR) 54(2):1\u201338","journal-title":"ACM Comput Surv (CSUR)"},{"key":"9197_CR88","first-page":"227","volume":"2018","author":"S Wen","year":"2018","unstructured":"Wen S, Kurc TM, Hou L et al (2018) Comparison of different classifiers with active learning to support quality control in nucleus segmentation in pathology images. AMIA Summits Transl Sci Proc 2018:227","journal-title":"AMIA Summits Transl Sci Proc"},{"key":"9197_CR89","volume-title":"Advances in neural information processing systems","author":"L Xu","year":"2019","unstructured":"Xu L, Skoularidou M, Cuesta-Infante A et al (2019) Modeling tabular data using conditional GAN. In: Wallach H, Larochelle H, Beygelzimer A et al (eds) Advances in neural information processing systems, vol 32. Curran Associates Inc, London"},{"issue":"4","key":"9197_CR90","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1145\/3328485","volume":"26","author":"W Xu","year":"2019","unstructured":"Xu W (2019) Toward human-centered AI: a perspective from human\u2013computer interaction. Interactions 26(4):42\u201346. https:\/\/doi.org\/10.1145\/3328485","journal-title":"Interactions"},{"issue":"1","key":"9197_CR91","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1109\/JBHI.2021.3088866","volume":"26","author":"B Zhan","year":"2022","unstructured":"Zhan B, Li D, Wu X et al (2022) Multi-modal MRI image synthesis via GAN with multi-scale gate mergence. IEEE J Biomed Health Inform 26(1):17\u201326. https:\/\/doi.org\/10.1109\/JBHI.2021.3088866","journal-title":"IEEE J Biomed Health Inform"},{"issue":"1","key":"9197_CR92","doi-asserted-by":"publisher","first-page":"780","DOI":"10.1109\/TVCG.2021.3114837","volume":"28","author":"Z Zhao","year":"2022","unstructured":"Zhao Z, Xu P, Scheidegger C et al (2022) Human-in-the-loop extraction of interpretable concepts in deep learning models. IEEE Trans Vis Comput Graph 28(1):780\u2013790. https:\/\/doi.org\/10.1109\/TVCG.2021.3114837","journal-title":"IEEE Trans Vis Comput Graph"},{"key":"9197_CR93","doi-asserted-by":"crossref","unstructured":"Zhu JY, Park T, Isola P et\u00a0al (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223\u20132232","DOI":"10.1109\/ICCV.2017.244"},{"key":"9197_CR94","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1007\/s00371-021-02322-z","volume":"39","author":"H Zhuang","year":"2021","unstructured":"Zhuang H, Zhang J, Liao F (2021) A systematic review on application of deep learning in digestive system image processing. Vis Comput 39:46\u201351. https:\/\/doi.org\/10.1007\/s00371-021-02322-z","journal-title":"Vis Comput"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-023-09197-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-023-09197-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-023-09197-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,19]],"date-time":"2024-01-19T15:08:00Z","timestamp":1705676880000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-023-09197-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,21]]},"references-count":94,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,2]]}},"alternative-id":["9197"],"URL":"https:\/\/doi.org\/10.1007\/s00521-023-09197-2","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,21]]},"assertion":[{"value":"21 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 October 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest regarding the publication of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}