{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T19:43:18Z","timestamp":1779219798150,"version":"3.51.4"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,1,22]],"date-time":"2020-01-22T00:00:00Z","timestamp":1579651200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,1,22]],"date-time":"2020-01-22T00:00:00Z","timestamp":1579651200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["K\u00fcnstl Intell"],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This article discusses the counterpart of interactive machine learning, i.e., human learning while being in the loop in a human-machine collaboration. For such cases we propose the use of a Contradiction Matrix to assess the overlap and the contradictions of human and machine predictions. We show in a small-scaled user study with experts in the area of pneumology (1) that machine-learning based systems can classify X-rays with respect to diseases with a meaningful accuracy, (2) humans partly use contradictions to reconsider their initial diagnosis, and (3) that this leads to a higher overlap between human and machine diagnoses at the end of the collaboration situation. We argue that disclosure of information on diagnosis uncertainty can be beneficial to make the human expert reconsider her or his initial assessment which may ultimately result in a deliberate agreement. In the light of the observations from our project, it becomes apparent that collaborative learning in such a human-in-the-loop scenario could lead to mutual benefits for both human learning and interactive machine learning. Bearing the differences in reasoning and learning processes of humans and intelligent systems in mind, we argue that interdisciplinary research teams have the best chances at tackling this undertaking and generating valuable insights.<\/jats:p>","DOI":"10.1007\/s13218-020-00638-x","type":"journal-article","created":{"date-parts":[[2020,1,22]],"date-time":"2020-01-22T08:03:10Z","timestamp":1579680190000},"page":"199-207","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["How and What Can Humans Learn from Being in the Loop?"],"prefix":"10.1007","volume":"34","author":[{"given":"Benjamin M.","family":"Abdel-Karim","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicolas","family":"Pfeuffer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gernot","family":"Rohde","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4757-0599","authenticated-orcid":false,"given":"Oliver","family":"Hinz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,1,22]]},"reference":[{"issue":"4","key":"638_CR1","first-page":"105","volume":"35","author":"S Amershi","year":"2014","unstructured":"Amershi S, Cakmak M, Knox WB, Kulesza T (2014) Power to the people: the role of humans in interactive machine learning. Assoc Adv Artif Intell 35(4):105\u2013120","journal-title":"Assoc Adv Artif Intell"},{"issue":"4698","key":"638_CR2","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1126\/science.228.4698.456","volume":"228","author":"JR Anderson","year":"1985","unstructured":"Anderson JR, Boyle CF, Reiser BJ (1985) Intelligent tutoring systems. Science 228(4698):456\u2013462","journal-title":"Science"},{"issue":"3","key":"638_CR3","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1037\/h0032653","volume":"63","author":"RC Anderson","year":"1972","unstructured":"Anderson RC, Kulhavy RW, Andre T (1972) Conditions under which feedback facilitates learning from programmed lessons. J Educ Psychol 63(3):186","journal-title":"J Educ Psychol"},{"issue":"2","key":"638_CR4","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1037\/h0047507","volume":"58","author":"JM Barnes","year":"1959","unstructured":"Barnes JM, Underwood BJ (1959) \u201cfate\u201d of first-list associations in transfer theory. J Exp Psychol 58(2):97","journal-title":"J Exp Psychol"},{"issue":"4","key":"638_CR5","doi-asserted-by":"publisher","first-page":"883","DOI":"10.2307\/41409965","volume":"34","author":"P Bera","year":"2011","unstructured":"Bera P, Burton-Jones A, Wand Y (2011) Guidelines for designing visual ontologies to support knowledge identification. MIS Q 34(4):883\u2013908","journal-title":"MIS Q"},{"key":"638_CR6","doi-asserted-by":"crossref","unstructured":"Bussone A, Stumpf S, O\u2019Sullivan D (2015) The role of explanations on trust and reliance in clinical decision support systems. Healthcare informatics (ICHI). In: IEEE International Conference on healthcate informatics. Dallas, pp 160\u2013169","DOI":"10.1109\/ICHI.2015.26"},{"issue":"4","key":"638_CR7","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1111\/j.1467-9280.2006.01704.x","volume":"17","author":"N Epley","year":"2006","unstructured":"Epley N, Gilovich T (2006) The anchoring-and-adjustment heuristic: why the adjustments are insufficient. Psychol Sci 17(4):311\u2013318","journal-title":"Psychol Sci"},{"issue":"2","key":"638_CR8","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(2):115\u2013127","journal-title":"Nature"},{"key":"638_CR9","first-page":"81","volume-title":"Advances in intelligent tutoring systems. Studies in computational intelligence","author":"P Fournier-Viger","year":"2010","unstructured":"Fournier-Viger P, Nkambou R, Nguifo EM (2010) Building intelligent tutoring systems for ill-defined domains. In: Nkambou R, Bourdeau J, Mizoguchi R (eds) Advances in intelligent tutoring systems. Studies in computational intelligence, vol 308. Springer, Berlin, Heidelberg, pp 81\u2013101"},{"issue":"4","key":"638_CR10","doi-asserted-by":"publisher","first-page":"497","DOI":"10.2307\/249487","volume":"23","author":"S Gregor","year":"1999","unstructured":"Gregor S, Benbasat I (1999) Explanations from intelligent systems: theoretical foundations and implications for practice. MISQ 23(4):497\u2013530","journal-title":"MIS Q"},{"key":"638_CR11","doi-asserted-by":"crossref","unstructured":"Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Marklund H, Haghgoo B, Ball R, Shpanskaya K, Seekins J, Mong DA, Halabi SS, Sandberg JK, Jones R, Larson DB, Langlotz CP, Patel BN, Lungren MP, Ng AY (2019) Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: Association for the advancement of artificial intelligence,\u00a0pp 1\u20139","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"638_CR12","first-page":"1097","volume-title":"Advances in neural information processing systems","author":"A Krizhevsky","year":"2012","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems. Springer, Berlin, pp 1097\u20131105"},{"issue":"3","key":"638_CR13","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1037\/0022-0663.79.3.320","volume":"79","author":"KG Lhyle","year":"1987","unstructured":"Lhyle KG, Kulhavy RW (1987) Feedback processing and error correction. J Educ Psychol 79(3):320","journal-title":"J Educ Psychol"},{"issue":"1","key":"638_CR14","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1016\/S0957-4174(01)00046-X","volume":"22","author":"S hsien Liao","year":"2002","unstructured":"hsien Liao S (2002) Problem solving and knowledge inertia. Expert Syst Appl 22(1):21\u201331","journal-title":"Expert Syst Appl"},{"key":"638_CR15","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/THFE2.1960.4503259","volume":"1","author":"JCR Licklider","year":"1960","unstructured":"Licklider JCR (1960) Man-computer symbiosis. IRE Trans Hum Factors Electron 1:4\u201311","journal-title":"IRE Trans Hum Factors Electron"},{"issue":"4","key":"638_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s12599-019-00600-8","volume":"61","author":"A Maedche","year":"2019","unstructured":"Maedche A, Legner C, Benlian A, Berger B, Gimpel H, Hess T, Hinz O, Morana S, S\u00f6llner M (2019) Ai-based digital assistants. Bus Inf Syst Eng 61(4):1\u201310","journal-title":"Bus Inf Syst Eng"},{"issue":"3","key":"638_CR17","first-page":"500","volume":"64","author":"G Marchant","year":"1989","unstructured":"Marchant G (1989) Analogical reasoning and hypothesis generation in auditing. Account Rev 64(3):500\u2013513","journal-title":"Account Rev"},{"issue":"2017","key":"638_CR18","first-page":"456","volume":"68","author":"J Metcalfe","year":"2017","unstructured":"Metcalfe J (2017) Learning from errors. Annu Rev Psychol 68(2017):456\u2013489","journal-title":"Annu Rev Psychol"},{"issue":"2","key":"638_CR19","doi-asserted-by":"publisher","first-page":"225","DOI":"10.3758\/BF03194056","volume":"14","author":"J Metcalfe","year":"2007","unstructured":"Metcalfe J, Kornell N (2007) Principles of cognitive science in education: the effects of generation, errors, and feedback. Psychon Bul Rev 14(2):225\u2013229","journal-title":"Psychon Bul Rev"},{"issue":"2","key":"638_CR20","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1037\/0033-295X.115.2.502","volume":"115","author":"DA Moore","year":"2008","unstructured":"Moore DA, Healy PJ (2008) The trouble with overconfidence. Psychol Rev 115(2):502\u2013517","journal-title":"Psychol Rev"},{"issue":"1","key":"638_CR21","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1207\/s15327108ijap0801_3","volume":"8","author":"KL Mosier","year":"1997","unstructured":"Mosier KL, Skitka LJ, Heers S, Burdick M (1997) Automation bias: decision making and performance in high-tech cockpits. Int J Aviat Psychol 8(1):47\u201363","journal-title":"Int J Aviat Psychol"},{"issue":"1","key":"638_CR22","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1016\/j.ijresmar.2006.02.001","volume":"23","author":"EJ Nijssen","year":"2006","unstructured":"Nijssen EJ, Hillebr B, Vermeulen PA, Kemp RG (2006) Exploring product and service innovation similarities and differences. Int J Res Mark 23(1):241\u2013251","journal-title":"Int J Res Mark"},{"issue":"4","key":"638_CR23","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1007\/s12599-019-00599-y","volume":"61","author":"N Pfeuffer","year":"2019","unstructured":"Pfeuffer N, Benlian A, Gimpel H, Hinz O (2019) Anthropomorphic information systems. Bus Inf Syst Eng 61(4):523\u2013533","journal-title":"Bus Inf Syst Eng"},{"issue":"1","key":"638_CR24","doi-asserted-by":"publisher","first-page":"21","DOI":"10.2307\/41410404","volume":"36","author":"GL Polites","year":"2012","unstructured":"Polites GL, Karahanna E (2012) Shackled to the status quo: the inhibiting effects of incumbent system habit, switching costs, and inertia on new system acceptance. MIS Q 36(1):21\u201342","journal-title":"MIS Q"},{"issue":"1","key":"638_CR25","first-page":"1","volume":"11","author":"P Rajpurkar","year":"2017","unstructured":"Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, Ding Daisy, Bagul Aarti, Langlotz C, Shpanskaya K, Lungren MP, Ng AY (2017) Chexnet: radiologist-level pneumonia detection on chest X-rays with deep learning. Comput Vis Pattern Recognit 11(1):1\u20137","journal-title":"Comput Vis Pattern Recognit"},{"key":"638_CR26","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1007\/978-3-319-90403-0_3","volume-title":"Human and machine learning: visible, explainable, trustworthy and transparent","author":"PC Shih","year":"2018","unstructured":"Shih PC (2018) Beyond human-in-the-loop: empowering end-users with transparent machine learning. In: Jianlong Z, Fang C (eds) Human and machine learning: visible, explainable, trustworthy and transparent. Springer, Berlin, pp 37\u201357"},{"issue":"1","key":"638_CR27","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1016\/j.cose.2018.09.012","volume":"80","author":"M Sillic","year":"2019","unstructured":"Sillic M (2019) Critical impact of organizational and individual inertia in explaining non-compliant security behavior in the shadow it context. Comput Secur 80(1):108\u2013119","journal-title":"Comput Secur"},{"issue":"529","key":"638_CR28","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","volume":"484","author":"D Silver","year":"2016","unstructured":"Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of go with deep neural networks and tree search. Nature 484(529):484\u2013489","journal-title":"Nature"},{"issue":"10","key":"638_CR29","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1038\/nature24270","volume":"550","author":"D Silver","year":"2017","unstructured":"Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y, Lillicrap T, Hui F, Sifre L, van den Driessche G, Graepel T, Hassabis D (2017) Mastering the game of go without human knowledge. Nature 550(10):354\u2013359","journal-title":"Nature"},{"key":"638_CR30","volume-title":"Science and Human Behavior. 92904","author":"BF Skinner","year":"1965","unstructured":"Skinner BF (1965) Science and Human Behavior. 92904. Simon and Schuster, New York"},{"issue":"1","key":"638_CR31","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1016\/S0034-4257(97)00083-7","volume":"62","author":"SV Stehman","year":"1997","unstructured":"Stehman SV (1997) Selecting and interpreting measures of thematic classification accuracy. Remote Sens Environ 62(1):77\u201389","journal-title":"Remote Sens Environ"},{"key":"638_CR32","volume-title":"Learning gap: why our schools are failing and what we can learn from Japanese and Chinese education","author":"H Stevenson","year":"1994","unstructured":"Stevenson H, Stigler JW (1994) Learning gap: why our schools are failing and what we can learn from Japanese and Chinese education. Simon and Schuster, New York"},{"issue":"12","key":"638_CR33","first-page":"1","volume":"11","author":"C Szegedy","year":"2015","unstructured":"Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2015) Rethinking the inception architecture for computer vision. Comput Vis Pattern Recognit 11(12):1\u201310","journal-title":"Comput Vis Pattern Recognit"},{"issue":"4","key":"638_CR34","doi-asserted-by":"publisher","first-page":"493","DOI":"10.2307\/248979","volume":"11","author":"P Todd","year":"1987","unstructured":"Todd P, Benbasat I (1987) Process tracing methods in decision support systems research: exploring the black box. MIS Q 11(4):493\u2013512","journal-title":"MIS Q"},{"issue":"4","key":"638_CR35","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1080\/00461520.2011.611369","volume":"46","author":"K VanLehn","year":"2011","unstructured":"VanLehn K (2011) The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol 46(4):197\u2013221. https:\/\/doi.org\/10.1080\/00461520.2011.611369","journal-title":"Educ Psychol"},{"issue":"3","key":"638_CR36","doi-asserted-by":"publisher","first-page":"221","DOI":"10.2307\/248950","volume":"9","author":"NP Vitalari","year":"1985","unstructured":"Vitalari NP (1985) Knowledge as a basis for expertise in systems analysis: an empirical study. MIS Q 9(3):221\u2013241","journal-title":"MIS Q"},{"issue":"9","key":"638_CR37","first-page":"2097","volume":"27","author":"X Wang","year":"2017","unstructured":"Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM (2017) Chestx-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. IEEE CVPR 2017 27(9):2097\u20132106","journal-title":"IEEE CVPR 2017"},{"issue":"1","key":"638_CR38","first-page":"436","volume":"521","author":"L Yann","year":"2015","unstructured":"Yann L, Yoshua B, Geoffrey H (2015) Deep learning. Nature 521(1):436\u2013444","journal-title":"Nature"},{"issue":"1","key":"638_CR39","first-page":"1","volume":"10","author":"L Yao","year":"2017","unstructured":"Yao L, Poblenz E, Dagunts D, Covington B, Bernard D, Lyman K (2017) Learning to diagnose from scratch by exploiting dependencies among labels. Comput Vis Pattern Recognit 10(1):1\u201312","journal-title":"Comput Vis Pattern Recognit"}],"container-title":["KI - K\u00fcnstliche Intelligenz"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00638-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s13218-020-00638-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s13218-020-00638-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,1,21]],"date-time":"2021-01-21T00:40:27Z","timestamp":1611189627000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s13218-020-00638-x"}},"subtitle":["Invoking Contradiction Learning as a Measure to Make Humans Smarter"],"short-title":[],"issued":{"date-parts":[[2020,1,22]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["638"],"URL":"https:\/\/doi.org\/10.1007\/s13218-020-00638-x","relation":{},"ISSN":["0933-1875","1610-1987"],"issn-type":[{"value":"0933-1875","type":"print"},{"value":"1610-1987","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,1,22]]},"assertion":[{"value":"16 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 January 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 January 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}