{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T11:28:40Z","timestamp":1764588520307,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,5,9]],"date-time":"2024-05-09T00:00:00Z","timestamp":1715212800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"UNESCO Chair on AI &amp; VR","award":["2022.09212.PTDC"],"award-info":[{"award-number":["2022.09212.PTDC"]}]},{"DOI":"10.13039\/501100001871","name":"national funds through Funda\u00e7\u00e3o para a Ci\u00eancia e a Tecnologia","doi-asserted-by":"publisher","award":["2022.09212.PTDC"],"award-info":[{"award-number":["2022.09212.PTDC"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Integrating eye gaze data with chest X-ray images in deep learning (DL) has led to contradictory conclusions in the literature. Some authors assert that eye gaze data can enhance prediction accuracy, while others consider eye tracking irrelevant for predictive tasks. We argue that this disagreement lies in how researchers process eye-tracking data as most remain agnostic to the human component and apply the data directly to DL models without proper preprocessing. We present EyeXNet, a multimodal DL architecture that combines images and radiologists\u2019 fixation masks to predict abnormality locations in chest X-rays. We focus on fixation maps during reporting moments as radiologists are more likely to focus on regions with abnormalities and provide more targeted regions to the predictive models. Our analysis compares radiologist fixations in both silent and reporting moments, revealing that more targeted and focused fixations occur during reporting. Our results show that integrating the fixation masks in a multimodal DL architecture outperformed the baseline model in five out of eight experiments regarding average Recall and six out of eight regarding average Precision. Incorporating fixation masks representing radiologists\u2019 classification patterns in a multimodal DL architecture benefits lesion detection in chest X-ray (CXR) images, particularly when there is a strong correlation between fixation masks and generated proposal regions. This highlights the potential of leveraging fixation masks to enhance multimodal DL architectures for CXR image analysis. This work represents a first step towards human-centered DL, moving away from traditional data-driven and human-agnostic approaches.<\/jats:p>","DOI":"10.3390\/make6020048","type":"journal-article","created":{"date-parts":[[2024,5,9]],"date-time":"2024-05-09T05:16:45Z","timestamp":1715231805000},"page":"1055-1071","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3352-6899","authenticated-orcid":false,"given":"Chihcheng","family":"Hsieh","sequence":"first","affiliation":[{"name":"School of Information Systems, Queensland University of Technology, Brisbane, QLD 4000, Australia"}]},{"given":"Andr\u00e9","family":"Lu\u00eds","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal"},{"name":"INESC-ID, 1000-029 Lisbon, Portugal"}]},{"given":"Jos\u00e9","family":"Neves","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal"},{"name":"INESC-ID, 1000-029 Lisbon, Portugal"}]},{"given":"Isabel Blanco","family":"Nobre","sequence":"additional","affiliation":[{"name":"Grupo Lus\u00edadas, Imagiology Department, 1500-458 Lisbon, Portugal"}]},{"given":"Sandra Costa","family":"Sousa","sequence":"additional","affiliation":[{"name":"Grupo Lus\u00edadas, Imagiology Department, 1500-458 Lisbon, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7098-5480","authenticated-orcid":false,"given":"Chun","family":"Ouyang","sequence":"additional","affiliation":[{"name":"School of Information Systems, Queensland University of Technology, Brisbane, QLD 4000, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5441-4637","authenticated-orcid":false,"given":"Joaquim","family":"Jorge","sequence":"additional","affiliation":[{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal"},{"name":"INESC-ID, 1000-029 Lisbon, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8826-5163","authenticated-orcid":false,"given":"Catarina","family":"Moreira","sequence":"additional","affiliation":[{"name":"School of Information Systems, Queensland University of Technology, Brisbane, QLD 4000, Australia"},{"name":"Instituto Superior T\u00e9cnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal"},{"name":"Human Technology Institute, University of Technology Sydney, Sydney, NSW 2007, Australia"}]}],"member":"1968","published-online":{"date-parts":[[2024,5,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"S34","DOI":"10.2214\/AJR.07.7081","article-title":"Radiologic Signs in Thoracic Imaging: Case-Based Review and Self-Assessment Module","volume":"192","author":"Parker","year":"2009","journal-title":"Am. J. Roentgenol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1111\/1754-9485.13273","article-title":"Deep learning applied to automatic disease detection using chest X-rays","volume":"65","author":"Moses","year":"2021","journal-title":"J. Med. Imaging Radiat. Oncol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1038\/s41597-022-01441-z","article-title":"REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays","volume":"9","author":"Zhang","year":"2022","journal-title":"Sci. Data"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1038\/s41597-021-00863-5","article-title":"Creation and validation of a chest X-ray dataset with eye-tracking and report dictation for AI development","volume":"8","author":"Karargyris","year":"2021","journal-title":"Sci. Data"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Lu\u00eds, A., Hsieh, C., Nobre, I.B., Sousa, S.C., Maciel, A., Moreira, C., and Jorge, J. (2023, January 25\u201329). Integrating Eye-Gaze Data into CXR DL Approaches: A Preliminary study. Proceedings of the 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Shanghai, China.","DOI":"10.1109\/VRW58643.2023.00048"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"767","DOI":"10.1007\/s10278-022-00760-2","article-title":"Changes in Radiologists\u2019 Gaze Patterns Against Lung X-rays with Different Abnormalities: A Randomized Experiment","volume":"36","author":"Pershin","year":"2023","journal-title":"J. Digit. Imaging"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Castner, N., Kuebler, T.C., Scheiter, K., Richter, J., Eder, T., H\u00fcttig, F., Keutel, C., and Kasneci, E. (2020, January 2\u20135). Deep semantic gaze embedding and scanpath comparison for expertise classification during OPT viewing. Proceedings of the ACM Symposium on Eye Tracking Research and Applications, Stuttgart, Germany.","DOI":"10.1145\/3379155.3391320"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1038\/s42256-022-00536-x","article-title":"Benchmarking saliency methods for chest X-ray interpretation","volume":"4","author":"Saporta","year":"2022","journal-title":"Nat. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"111341","DOI":"10.1016\/j.ejrad.2024.111341","article-title":"Shedding light on ai in radiology: A systematic review and taxonomy of eye gaze-driven interpretability in deep learning","volume":"172","author":"Neves","year":"2024","journal-title":"Eur. J. Radiol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1109\/TMI.2022.3146973","article-title":"Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis","volume":"41","author":"Wang","year":"2022","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., and Essert, C. (October, January 27). Observational Supervision for Medical Image Classification Using Gaze Data. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2021, Strasbourg, France.","DOI":"10.1007\/978-3-030-87196-3"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"991683","DOI":"10.3389\/fradi.2022.991683","article-title":"Improving disease classification performance and explainability of deep learning models in radiology with heatmap generators","volume":"2","author":"Watanabe","year":"2022","journal-title":"Front. Radiol."},{"key":"ref_13","unstructured":"Nie, W., Zhang, Y., and Patel, A. (2018). A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations. arXiv."},{"key":"ref_14","unstructured":"Agnihotri, P., Ketabi, S., and Khalvati, F. (2022). Using Multi-modal Data for Improving Generalizability and Explainability of Disease Classification in Radiology. arXiv."},{"key":"ref_15","unstructured":"Qi, Z., Khorram, S., and Li, F. (2019). Visualizing Deep Networks by Optimizing with Integrated Gradients. arXiv."},{"key":"ref_16","unstructured":"Lanfredi, R.B., Arora, A., Drew, T., Schroeder, J.D., and Tasdizen, T. (2021). Comparing radiologists\u2019 gaze and saliency maps generated by interpretability methods for chest X-rays. arXiv."},{"key":"ref_17","unstructured":"Moreira, C., Alvito, D., Sousa, S.C., Nobre, I.B., Ouyang, C., Kopper, R., Duchowski, A., and Jorge, J. (June, January 29). Comparing Visual Search Patterns in Chest X-ray Diagnostics. Proceedings of the ACM on Computer Graphics and Interactive Techniques (ETRA), T\u00fcbingen, Germany."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Shneiderman, B. (2022). Human-Centered AI, Oxford University Press.","DOI":"10.1093\/oso\/9780192845290.001.0001"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"4459198","DOI":"10.1155\/2023\/4459198","article-title":"Towards Risk-Free Trustworthy Artificial Intelligence: Significance and Requirements","volume":"2023","author":"Alzubaidi","year":"2023","journal-title":"Int. J. Intell. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1145\/3458652","article-title":"Medical artificial intelligence: The European legal perspective","volume":"64","author":"Schneeberger","year":"2021","journal-title":"Commun. ACM"},{"key":"ref_21","unstructured":"Holmqvist, K., and Andersson, R. (2017). Eye-Tracking: A Comprehensive Guide to Methods, Paradigms and Measures, Oxford University Press."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1241","DOI":"10.1148\/radiographics.7.6.3423330","article-title":"Using eye movements to study visual search and to improve tumor detection","volume":"7","author":"Nodine","year":"1987","journal-title":"RadioGraphics"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Duchowski, A.T. (2003). Eye Tracking Methodology: Theory and Practice, Springer.","DOI":"10.1007\/978-1-4471-3750-4"},{"key":"ref_24","unstructured":"Rong, Y., Xu, W., Akata, Z., and Kasneci, E. (2021, January 22\u201325). Human Attention in Fine-grained Classification. Proceedings of the 32nd British Machine Vision Conference (BMVC), Online."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Woo, S., Park, J., Lee, J., and Kweon, I.S. (2018). CBAM: Convolutional Block Attention Module. arXiv.","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"ref_27","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An Image is Worth 16 \u00d7 16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"594","DOI":"10.3758\/BF03207377","article-title":"Finding lung nodules with and without comparative visual scanning","volume":"29","author":"Carmody","year":"1981","journal-title":"Percept. Psychophys."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/S1076-6332(05)80381-2","article-title":"Visual scanning patterns of radiologists searching mammograms","volume":"3 2","author":"Krupinski","year":"1996","journal-title":"Academic radiology"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/S1076-6332(05)80780-9","article-title":"Searching for bone fractures: A comparison with pulmonary nodule search","volume":"1","author":"Hu","year":"1994","journal-title":"Acad. Radiol."},{"key":"ref_31","unstructured":"Kendall, A., Gal, Y., and Cipolla, R. (2017). Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Matthew Zeiler, D., and Rob, F. (2014, January 6\u201312). Visualizing and understanding convolutional neural networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_33","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.243"},{"key":"ref_36","unstructured":"Tan, M., and Le, Q. (2019, January 10\u201315). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning. PMLR, Long Beach, CA, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18\u201324). A convnet for the 2020s. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.01167"},{"key":"ref_38","first-page":"568","article-title":"Two-stream convolutional networks for action recognition in videos","volume":"27","author":"Simonyan","year":"2014","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., and Doll\u00e1r, P. (2020, January 13\u201319). Designing network design spaces. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01044"},{"key":"ref_40","unstructured":"Hsieh, C., Ouyang, C., Nascimento, J.C., Pereira, J.a., Jorge, J., and Moreira, C. (2023). MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multimodal Deep Learning Applications (version 1.0.0). PhysioNet."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1038\/s41597-019-0322-0","article-title":"MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports","volume":"6","author":"Johnson","year":"2019","journal-title":"Sci. Data"},{"key":"ref_42","unstructured":"Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., and Horng, S. (2019). MIMIC-CXR Database (version 2.0.0). PhysioNet."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Johnson, A.E., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., and Horng, S. (2019). MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv.","DOI":"10.1038\/s41597-019-0322-0"},{"key":"ref_44","unstructured":"Johnson, A., Bulgarelli, L., Pollard, T., Celi, L.A., Mark, R., and Horng IV, S. (2023). MIMIC-IV-ED (Version: 2.2). PhysioNet."},{"key":"ref_45","unstructured":"Johnson, A., Bulgarelli, L., Pollard, T., Horng, S., Celi, L.A., and Mark, R. (2023). MIMIC-IV (version 2.2). Physionet."},{"key":"ref_46","first-page":"e215","article-title":"PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2020","journal-title":"Circulation"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1016\/j.acra.2013.03.001","article-title":"A brief history of free-response receiver operating characteristic paradigm data analysis","volume":"20","author":"Chakraborty","year":"2013","journal-title":"Acad. Radiol."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1111\/1754-9485.12798","article-title":"A review of factors influencing radiologists\u2019 visual search behaviour","volume":"62","author":"Ganesan","year":"2018","journal-title":"J. Med. Imaging Radiat. Oncol."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Hsieh, C., Nobre, I.B., Sousa, S.C., Ouyang, C., Brereton, M., Nascimento, J.C., Jorge, J., and Moreira, C. (2023). MDF-Net: Multimodal Dual-Fusion Network for Abnormality Detection using CXR Images and Clinical Data. arXiv.","DOI":"10.1038\/s41598-023-41463-0"},{"key":"ref_50","first-page":"11","article-title":"Eye-tracking metrics in perception and visual attention research","volume":"3","author":"Borys","year":"2017","journal-title":"EJMT"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1016\/j.compmedimag.2017.04.006","article-title":"Application of eye tracking in medicine: A survey, research issues and challenges","volume":"65","author":"Harezlak","year":"2018","journal-title":"Comput. Med. Imaging Graph."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"035502","DOI":"10.1117\/1.JMI.5.3.035502","article-title":"Modeling visual search behavior of breast radiologists using a deep convolution neural network","volume":"5","author":"Mall","year":"2018","journal-title":"J. Med. Imaging"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1007\/s10278-018-00174-z","article-title":"Can a Machine Learn from Radiologists\u2019 Visual Search Behaviour and Their Interpretation of Mammograms\u2014A Deep-Learning Study","volume":"32","author":"Mall","year":"2019","journal-title":"J. Digit. Imaging"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/j.acra.2016.11.020","article-title":"Fixated and Not Fixated Regions of Mammograms, A Higher-Order Statistical Analysis of Visual Search Behavior","volume":"24","author":"Mall","year":"2017","journal-title":"Acad. Radiol."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.media.2018.10.010","article-title":"A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning","volume":"51","author":"Khosravan","year":"2019","journal-title":"Med. Image Anal."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/2\/48\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:42:34Z","timestamp":1760107354000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/6\/2\/48"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,9]]},"references-count":55,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["make6020048"],"URL":"https:\/\/doi.org\/10.3390\/make6020048","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2024,5,9]]}}}