{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T11:40:19Z","timestamp":1792496419918,"version":"4.5.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T00:00:00Z","timestamp":1742169600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T00:00:00Z","timestamp":1742169600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100004320","name":"Philips","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004320","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Despite the approval of over 200 artificial intelligence (AI) applications for radiology in the European Union, widespread adoption in clinical practice remains limited. Current assessments of AI applications often rely on post-hoc evaluations, lacking the granularity to capture real-time radiologist-AI interactions. The purpose of the study is to realise the Radiology AI lab for real-time, objective measurement of the impact of AI applications on radiologists\u2019 workflows. We proposed the user-state sensing framework (USSF) to structure the sensing of radiologist-AI interactions in terms of personal, interactional, and contextual states. Guided by the USSF, a lab was established using three non-invasive biometric measurement techniques: eye-tracking, heart rate monitoring, and facial expression analysis. We conducted a pilot test with four radiologists of varying experience levels, who read ultra-low-dose (ULD) CT cases in (1) standard PACS and (2) manually annotated (to mimic AI) PACS workflows. Interpretation time, eye-tracking metrics, heart rate variability (HRV), and facial expressions were recorded and analysed. The Radiology AI lab was successfully realised as an initial physical iteration of the USSF at a tertiary referral centre. Radiologists participating in the pilot test read 32 ULDCT cases (mean age, 52\u00a0years\u2009\u00b1\u200923 (SD); 17 male; 16 cases with abnormalities). Cases were read on average in 4.1\u2009\u00b1\u20092.2\u00a0min (standard PACS) and 3.9\u2009\u00b1\u20091.9\u00a0min (AI-annotated PACS), with no significant difference (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009=\u20090.48). Three out of four radiologists showed significant shifts (\n                    <jats:italic>p<\/jats:italic>\n                    \u2009&lt;\u20090.02) in eye-tracking metrics, including saccade duration, saccade quantity, fixation duration, fixation quantity, and pupil diameter, when using the AI-annotated workflow. These changes align with prior findings linking such metrics to increased competency and reduced cognitive load, suggesting a more efficient visual search strategy in AI-assisted interpretation. Although HRV metrics did not correlate with experience, when combined with facial expression analysis, they helped identify key moments during the pilot test. The Radiology AI lab was successfully realised, implementing personal, interactional, and contextual states of the user-state sensing framework, enabling objective analysis of radiologists\u2019 workflows, and effectively capturing relevant biometrics. Future work will focus on expanding sensing of the contextual state of the user-state sensing framework, refining baseline determination, and continuing investigation of AI-enabled tools in radiology workflows.\n                  <\/jats:p>","DOI":"10.1007\/s10278-025-01453-2","type":"journal-article","created":{"date-parts":[[2025,3,17]],"date-time":"2025-03-17T14:20:48Z","timestamp":1742221248000},"page":"4303-4311","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Radiology AI Lab: Evaluation of Radiology Applications with Clinical End-Users"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8790-7964","authenticated-orcid":false,"given":"Olivier","family":"Paalvast","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Merlijn","family":"Sevenster","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omar","family":"Hertgers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hubrecht","family":"de Bliek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victor","family":"Wijn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vincent","family":"Buil","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jaap","family":"Knoester","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandra","family":"Vosbergen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hildo","family":"Lamb","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,3,17]]},"reference":[{"key":"1453_CR1","unstructured":"Van Leeuwen, K.G., et al., Comparison of Commercial AI Software Performance for Radiograph Lung Nodule Detection and Bone Age Prediction. Radiology, 2024. 310(1)."},{"key":"1453_CR2","doi-asserted-by":"crossref","unstructured":"Van Leeuwen, K.G., et al., Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. European Radiology, 2021. 31(6): p. 3797-3804.","DOI":"10.1007\/s00330-021-07892-z"},{"key":"1453_CR3","doi-asserted-by":"crossref","unstructured":"Fryback, D.G. and J.R. Thornbury, The Efficacy of Diagnostic Imaging. Medical Decision Making, 1991. 11(2): p. 88-94.","DOI":"10.1177\/0272989X9101100203"},{"key":"1453_CR4","doi-asserted-by":"crossref","unstructured":"Mello-Thoms, C. and C.A.B. Mello, Clinical applications of artificial intelligence in radiology. The British Journal of Radiology, 2023. 96(1150).","DOI":"10.1259\/bjr.20221031"},{"key":"1453_CR5","doi-asserted-by":"crossref","unstructured":"Zech, J.R., et al., Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine, 2018. 15(11): p. e1002683.","DOI":"10.1371\/journal.pmed.1002683"},{"key":"1453_CR6","doi-asserted-by":"crossref","unstructured":"Cheung, J.L.S., et al., U\u201cAI\u201d Testing: User Interface and Usability Testing of a Chest X-ray AI Tool in a Simulated Real-World Workflow. Canadian Association of Radiologists Journal, 2022\u201310\u201302. 74(2).","DOI":"10.1177\/08465371221131200"},{"key":"1453_CR7","doi-asserted-by":"crossref","unstructured":"Yacoub, B., et al., Impact of Artificial Intelligence Assistance on Chest CT Interpretation Times: A Prospective Randomized Study. American Journal of Roentgenology, 2022-06-15. 219(5).","DOI":"10.2214\/AJR.22.27598"},{"key":"1453_CR8","doi-asserted-by":"crossref","unstructured":"Becker, C.D., et al., Current practical experience with artificial intelligence in clinical radiology: a survey of the European Society of Radiology. Insights into Imaging, 2022. 13(1).","DOI":"10.1186\/s13244-022-01247-y"},{"key":"1453_CR9","doi-asserted-by":"crossref","unstructured":"Brady, A.P., et al., Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA. Insights into Imaging, 2024. 15(1).","DOI":"10.1186\/s13244-023-01541-3"},{"key":"1453_CR10","doi-asserted-by":"crossref","unstructured":"Jaspers, M.W.M., A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International journal of medical informatics., 2009. 78(5): p. 340-353.","DOI":"10.1016\/j.ijmedinf.2008.10.002"},{"key":"1453_CR11","unstructured":"Lazar, J., J.H. Feng, and H. Hochheiser, Research Methods in Human-Computer Interaction. 2010: Wiley Publishing."},{"key":"1453_CR12","doi-asserted-by":"crossref","unstructured":"Jaspers, M.W., A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence - PubMed. International journal of medical informatics, 2009 May. 78(5).","DOI":"10.1016\/j.ijmedinf.2008.10.002"},{"key":"1453_CR13","doi-asserted-by":"crossref","unstructured":"Gegenfurtner, A., E. Lehtinen, and R. S\u00e4lj\u00f6, Expertise Differences in the Comprehension of Visualizations: a Meta-Analysis of Eye-Tracking Research in Professional Domains. Educational Psychology Review, 2011. 23(4): p. 523-552.","DOI":"10.1007\/s10648-011-9174-7"},{"key":"1453_CR14","doi-asserted-by":"crossref","unstructured":"Kok, E.M., et al., Looking in the Same Manner but Seeing it Differently: Bottom\u2010up and Expertise Effects in Radiology. Applied Cognitive Psychology, 2012. 26(6): p. 854-862.","DOI":"10.1002\/acp.2886"},{"key":"1453_CR15","doi-asserted-by":"crossref","unstructured":"L\u00e9v\u00eaque, L., P. Young, and H. Liu, Studying the Gaze Patterns of Expert Radiologists in Screening Mammography: A Case Study with Breast Test Wales | IEEE Conference Publication | IEEE Xplore. 2020 28th European Signal Processing Conference (EUSIPCO), 2021.","DOI":"10.23919\/Eusipco47968.2020.9287678"},{"key":"1453_CR16","doi-asserted-by":"crossref","unstructured":"Giovinco, N.A., et al., A Passing Glance? Differences in Eye Tracking and Gaze Patterns Between Trainees and Experts Reading Plain Film Bunion Radiographs. The Journal of Foot and Ankle Surgery, 2015\/05\/01. 54(3).","DOI":"10.1053\/j.jfas.2014.08.013"},{"key":"1453_CR17","doi-asserted-by":"crossref","unstructured":"Biondi, F.N., et al., Distracted worker: Using pupil size and blink rate to detect cognitive load during manufacturing tasks. Appl Ergon, 2023. 106: p. 103867.","DOI":"10.1016\/j.apergo.2022.103867"},{"key":"1453_CR18","doi-asserted-by":"crossref","unstructured":"Eltawil, F.A., et al., Analyzing Barriers and Enablers for the Acceptance of Artificial Intelligence Innovations into Radiology Practice: A Scoping Review. Tomography, 2023. 9(4): p. 1443-1455.","DOI":"10.3390\/tomography9040115"},{"key":"1453_CR19","doi-asserted-by":"crossref","unstructured":"Van Der Gijp, A., et al., How visual search relates to visual diagnostic performance: a narrative systematic review of eye-tracking research in radiology. Advances in Health Sciences Education, 2017. 22(3): p. 765-787.","DOI":"10.1007\/s10459-016-9698-1"},{"key":"1453_CR20","doi-asserted-by":"crossref","unstructured":"Rubin, G.D., et al., Characterizing Search, Recognition, and Decision in the Detection of Lung Nodules on CT Scans: Elucidation with Eye Tracking. Radiology, 2015. 274(1): p. 276-286.","DOI":"10.1148\/radiol.14132918"},{"key":"1453_CR21","doi-asserted-by":"crossref","unstructured":"Bertram, R., et al., The Effect of Expertise on Eye Movement Behaviour in Medical Image Perception. PLoS ONE, 2013. 8(6): p. e66169.","DOI":"10.1371\/journal.pone.0066169"},{"key":"1453_CR22","doi-asserted-by":"crossref","unstructured":"Solhjoo, S., et al., Heart Rate and Heart Rate Variability Correlate with Clinical Reasoning Performance and Self-Reported Measures of Cognitive Load. Scientific Reports, 2019. 9(1).","DOI":"10.1038\/s41598-019-50280-3"},{"key":"1453_CR23","doi-asserted-by":"crossref","unstructured":"American Heart, A., Heart Rate Variability. Circulation : the journal of the American Heart Association., 1996. 93(5): p. 1043-1065.","DOI":"10.1161\/01.CIR.93.5.1043"},{"key":"1453_CR24","doi-asserted-by":"crossref","unstructured":"Jaeger, S.R., et al., Valence, arousal and sentiment meanings of 33 facial emoji: Insights for the use of emoji in consumer research. Food Res Int, 2019. 119: p. 895-907.","DOI":"10.1016\/j.foodres.2018.10.074"},{"key":"1453_CR25","doi-asserted-by":"crossref","unstructured":"Shaffer, F. and J.P. Ginsberg, An Overview of Heart Rate Variability Metrics and Norms. Frontiers in Public Health, 2017. 5.","DOI":"10.3389\/fpubh.2017.00258"},{"key":"1453_CR26","doi-asserted-by":"crossref","unstructured":"Kauschke, C., et al., The Role of Emotional Valence for the Processing of Facial and Verbal Stimuli\u2014Positivity or Negativity Bias? Frontiers in Psychology, 2019. 10.","DOI":"10.3389\/fpsyg.2019.01654"}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-025-01453-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-025-01453-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-025-01453-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,12]],"date-time":"2025-12-12T19:24:13Z","timestamp":1765567453000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-025-01453-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,17]]},"references-count":26,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,12]]}},"alternative-id":["1453"],"URL":"https:\/\/doi.org\/10.1007\/s10278-025-01453-2","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,17]]},"assertion":[{"value":"20 September 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 February 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 February 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 March 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This is an observational study using retrospective patient data. Patient consent has been waived by METC-Leiden\u00a0Den Haag Delft for the use of patient data (number NL20210610001). Written informed consent was obtained from\u00a0participants in this study.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval and Consent to Participate"}},{"value":"Participants provided written consent for the use of their metrics in this publication.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}},{"value":"OP is appointed on a grant-subsided position in the TKI-LSH grant \u2018REPLAiCE\u2019 which as the Leiden University Medical\u00a0Center and Royal Philips B.V. as partners. OH is appointed on a subsidised position which received funding from\u00a0Royal Philips B.V. MS, HB, VW, VB, JK and SV are employed by Royal Philips B.V. HL is a consultant to Royal Philips\u00a0B.V.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}