{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,27]],"date-time":"2026-07-27T14:48:59Z","timestamp":1785163739804,"version":"3.55.0"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T00:00:00Z","timestamp":1779926400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,30]],"date-time":"2026-05-30T00:00:00Z","timestamp":1780099200000},"content-version":"vor","delay-in-days":2,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003252","name":"Lund University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003252","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Cancer"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The integration of clinical decision support tools in medical practice is challenging and must be carefully undertaken, especially in cancer management. Noninvasive Lymph Node Status (NILS) is a web-based tool designed to estimate the probability of benign axillary lymph nodes in female patients with breast cancer scheduled for primary surgery. The aim was to identify barriers to NILS adoption in a clinical setting and assess whether intended users can operate the tool without significant errors or difficulties. Additionally, the study aimed to evaluate the appropriateness of result interpretation for decisions to abstain from sentinel lymph node biopsy (SNLB) and measure overall user satisfaction with the tool.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This mixed-methods multicenter on-site qualitative human factor validation study was conducted in a simulated clinical environment, replicating both real-world physical and digital conditions. Based on the identified target user population for the NILS model, twenty physicians comprised the cohort. The study used simulated clinical cases, the System Usability Scale (SUS), and the After-Scenario Questionnaire (ASQ) to evaluate usability and satisfaction. An oral interview was conducted after the evaluations. The study followed a structured protocol, with distinct roles assigned to the test participants, leader, and observer.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Twenty physicians from four hospitals, with a median of 9.5 years of specialist practice, participated. Most participants were surgeons (75%,\n                      <jats:italic>N<\/jats:italic>\n                      \u2009=\u200915), while the remaining 25% (\n                      <jats:italic>N<\/jats:italic>\n                      \u2009=\u20095) were oncologists. Usability scores were high, with a mean SUS score of 89.5 (\u201cexcellent\u201d) and a mean ASQ score of 6.3 (Likert scale 1\u20137). Several interface challenges were identified, including difficulty in locating the reset and information buttons, the ability to enter values outside valid ranges (age and tumor size), and the risk of unintentionally modifying the entered values while scrolling.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>This study identified the key usability factors, barriers, and facilitators affecting the NILS model implementation. Physicians found the NILS interface easy to use and valued the presented risk estimates of benign axillary lymph node status in their decisions to perform or abstain from SLNB. Study-informed interface redesigns have been implemented. Iterative usability work in clinical settings is crucial for successful implementation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Trial registration<\/jats:title>\n                    <jats:p>ISRCTN99301435.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12885-026-16161-5","type":"journal-article","created":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T01:44:22Z","timestamp":1779932662000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Human factors validation study of an artificial neural network\u2011based preoperative decision\u2011support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN99301435)"],"prefix":"10.1186","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7889-9821","authenticated-orcid":false,"given":"Anna","family":"Allfelt","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8862-1845","authenticated-orcid":false,"given":"P\u00e4r-Ola","family":"Bendahl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7932-3982","authenticated-orcid":false,"given":"Looket","family":"Dihge","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1145-4297","authenticated-orcid":false,"given":"Mattias","family":"Ohlsson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7515-3130","authenticated-orcid":false,"given":"Lisa","family":"Ryd\u00e9n","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1265-8649","authenticated-orcid":false,"given":"Ida","family":"Skarping","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,28]]},"reference":[{"issue":"3","key":"16161_CR1","first-page":"209","volume":"71","author":"H Sung","year":"2021","unstructured":"Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F. Global cancer statistics 2020: GLOBOCAN Estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021;71(3):209\u201349.","journal-title":"CA Cancer J Clin"},{"issue":"3","key":"16161_CR2","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1001\/archsurg.1996.01430150079015","volume":"131","author":"B Cady","year":"1996","unstructured":"Cady B, Stone MD, Schuler JG, Thakur R, Wanner MA, Lavin PT. The new era in breast cancer. Invasion, size, and nodal involvement dramatically decreasing as a result of mammographic screening. Arch Surg. 1996;131(3):301\u20138.","journal-title":"Arch Surg"},{"issue":"9","key":"16161_CR3","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1093\/jnci\/djj158","volume":"98","author":"RE Mansel","year":"2006","unstructured":"Mansel RE, Fallowfield L, Kissin M, Goyal A, Newcombe RG, Dixon JM, Yiangou C, Horgan K, Bundred N, Monypenny I, et al. Randomized multicenter trial of sentinel node biopsy versus standard axillary treatment in operable breast cancer: the ALMANAC Trial. J Natl Cancer Inst. 2006;98(9):599\u2013609.","journal-title":"J Natl Cancer Inst"},{"issue":"6","key":"16161_CR4","doi-asserted-by":"publisher","first-page":"546","DOI":"10.1056\/NEJMoa012782","volume":"349","author":"U Veronesi","year":"2003","unstructured":"Veronesi U, Paganelli G, Viale G, Luini A, Zurrida S, Galimberti V, Intra M, Veronesi P, Robertson C, Maisonneuve P, et al. A randomized comparison of sentinel-node biopsy with routine axillary dissection in breast cancer. N Engl J Med. 2003;349(6):546\u201353.","journal-title":"N Engl J Med"},{"issue":"27","key":"16161_CR5","doi-asserted-by":"publisher","first-page":"3056","DOI":"10.1200\/JCO.21.00934","volume":"39","author":"M Brackstone","year":"2021","unstructured":"Brackstone M, Baldassarre FG, Perera FE, Cil T, Chavez Mac Gregor M, Dayes IS, Engel J, Horton JK, King TA, Kornecki A, et al. Management of the axilla in early-stage breast cancer: ontario health (Cancer Care Ontario) and ASCO guideline. J Clin Oncol. 2021;39(27):3056\u201382.","journal-title":"J Clin Oncol"},{"issue":"11","key":"16161_CR6","doi-asserted-by":"publisher","first-page":"1557","DOI":"10.1001\/jamaoncol.2023.3759","volume":"9","author":"OD Gentilini","year":"2023","unstructured":"Gentilini OD, Botteri E, Sangalli C, Galimberti V, Porpiglia M, Agresti R, Luini A, Viale G, Cassano E, Peradze N, et al. Sentinel lymph node biopsy vs no axillary surgery in patients with small breast cancer and negative results on ultrasonography of axillary lymph nodes: the SOUND randomized clinical trial. JAMA Oncol. 2023;9(11):1557\u201364.","journal-title":"JAMA Oncol"},{"key":"16161_CR7","doi-asserted-by":"crossref","unstructured":"Reimer T, Stachs A, Veselinovic K, K\u00fchn T, Heil J, Polata S, Marm\u00e9 F, M\u00fcller T, Hildebrandt G, Krug D, et al. Axillary surgery in breast cancer - primary results of the INSEMA Trial. N Engl J Med. 2025;392(11):1051\u201364.","DOI":"10.1056\/NEJMoa2412063"},{"key":"16161_CR8","doi-asserted-by":"publisher","first-page":"1102254","DOI":"10.3389\/fonc.2023.1102254","volume":"13","author":"L Dihge","year":"2023","unstructured":"Dihge L, Bendahl PO, Skarping I, Hj\u00e4rtstr\u00f6m M, Ohlsson M, Ryd\u00e9n L. The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer. Front Oncol. 2023;13:1102254.","journal-title":"Front Oncol"},{"issue":"10","key":"16161_CR9","doi-asserted-by":"publisher","first-page":"881","DOI":"10.1016\/S1470-2045(07)70278-4","volume":"8","author":"DN Krag","year":"2007","unstructured":"Krag DN, Anderson SJ, Julian TB, Brown AM, Harlow SP, Ashikaga T, Weaver DL, Miller BJ, Jalovec LM, Frazier TG, et al. Technical outcomes of sentinel-lymph-node resection and conventional axillary-lymph-node dissection in patients with clinically node-negative breast cancer: results from the NSABP B-32 randomised phase III trial. Lancet Oncol. 2007;8(10):881\u20138.","journal-title":"Lancet Oncol"},{"issue":"9","key":"16161_CR10","doi-asserted-by":"publisher","first-page":"2239","DOI":"10.1007\/s00268-012-1623-z","volume":"36","author":"S Pesek","year":"2012","unstructured":"Pesek S, Ashikaga T, Krag LE, Krag D. The false-negative rate of sentinel node biopsy in patients with breast cancer: a meta-analysis. World J Surg. 2012;36(9):2239\u201351.","journal-title":"World J Surg"},{"issue":"1","key":"16161_CR11","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1186\/s12885-019-5827-6","volume":"19","author":"L Dihge","year":"2019","unstructured":"Dihge L, Ohlsson M, Eden P, Bendahl PO, Ryden L. Artificial neural network models to predict nodal status in clinically node-negative breast cancer. BMC Cancer. 2019;19(1):610.","journal-title":"BMC Cancer"},{"key":"16161_CR12","doi-asserted-by":"publisher","first-page":"e46474","DOI":"10.2196\/46474","volume":"9","author":"M Hj\u00e4rtstr\u00f6m","year":"2023","unstructured":"Hj\u00e4rtstr\u00f6m M, Dihge L, Bendahl PO, Skarping I, Ellbrant J, Ohlsson M, Ryd\u00e9n L. Noninvasive staging of lymph node status in breast cancer using machine learning: external validation and further model development. JMIR Cancer. 2023;9:e46474.","journal-title":"JMIR Cancer"},{"issue":"1","key":"16161_CR13","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1186\/s12885-024-11854-1","volume":"24","author":"I Skarping","year":"2024","unstructured":"Skarping I, Ellbrant J, Dihge L, Ohlsson M, Huss L, Bendahl PO, Ryd\u00e9n L. Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750). BMC Cancer. 2024;24(1):86.","journal-title":"BMC Cancer"},{"issue":"3","key":"16161_CR14","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1007\/s10549-022-06636-x","volume":"194","author":"I Skarping","year":"2022","unstructured":"Skarping I, Nilsson K, Dihge L, Fridhammar A, Ohlsson M, Huss L, Bendahl PO, Steen Carlsson K, Ryd\u00e9n L. The implementation of a noninvasive lymph node staging (NILS) preoperative prediction model is cost effective in primary breast cancer. Breast Cancer Res Treat. 2022;194(3):577\u201386.","journal-title":"Breast Cancer Res Treat"},{"issue":"6","key":"16161_CR15","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1093\/intqhc\/mzm042","volume":"19","author":"A Tong","year":"2007","unstructured":"Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349\u201357.","journal-title":"Int J Qual Health Care"},{"key":"16161_CR16","unstructured":"U.S Food and Drug Administration. Applying human fctors and usability engineering to medical devices: guidance for industry and Food and Drug Administration Staff. Silver Spring (MD): FDA; 2016."},{"issue":"3","key":"16161_CR17","doi-asserted-by":"publisher","first-page":"379","DOI":"10.3758\/BF03195514","volume":"35","author":"L Faulkner","year":"2003","unstructured":"Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput. 2003;35(3):379\u201383.","journal-title":"Behav Res Methods Instrum Comput"},{"issue":"7","key":"16161_CR18","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1080\/10447318.2018.1455307","volume":"34","author":"JR Lewis","year":"2018","unstructured":"Lewis JR. The system usability scale: past, present, and future. Int J Human\u2013Computer Interact. 2018;34(7):577\u201390.","journal-title":"Int J Human\u2013Computer Interact"},{"issue":"8","key":"16161_CR19","doi-asserted-by":"publisher","first-page":"e37290","DOI":"10.2196\/37290","volume":"10","author":"M Hyzy","year":"2022","unstructured":"Hyzy M, Bond R, Mulvenna M, Bai L, Dix A, Leigh S, Hunt S. System usability scale benchmarking for digital health apps: meta-analysis. JMIR Mhealth Uhealth. 2022;10(8):e37290.","journal-title":"JMIR Mhealth Uhealth"},{"issue":"1","key":"16161_CR20","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1080\/10447319509526110","volume":"7","author":"JR Lewis","year":"1995","unstructured":"Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human\u2013Computer Interact. 1995;7(1):57\u201378.","journal-title":"Int J Human\u2013Computer Interact"},{"key":"16161_CR21","unstructured":"International Electrotechnical Commission (IEC). IEC TR 62366-2:2016 Medical devices - Part 2: Guidance on the application of usability engineering to medical devices (Edition 1.0) Geneva: IEC; 2016."},{"key":"16161_CR22","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/j.jbi.2018.01.001","volume":"78","author":"AL Russ","year":"2018","unstructured":"Russ AL, Saleem JJ. Ten factors to consider when developing usability scenarios and tasks for health information technology. J Biomed Inf. 2018;78:123\u201333.","journal-title":"J Biomed Inf"},{"issue":"194","key":"16161_CR23","first-page":"4","volume":"189","author":"J Brooke","year":"1996","unstructured":"Brooke J. SUS-A quick and dirty usability scale. Usability evaluation Ind. 1996;189(194):4\u20137.","journal-title":"Usability evaluation Ind"},{"issue":"3","key":"16161_CR24","first-page":"114","volume":"4","author":"A Bangor","year":"2009","unstructured":"Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J usability Stud. 2009;4(3):114\u201323.","journal-title":"J usability Stud"},{"issue":"15","key":"16161_CR25","doi-asserted-by":"publisher","first-page":"5141","DOI":"10.1002\/cam4.4072","volume":"10","author":"GD Farmer","year":"2021","unstructured":"Farmer GD, Pearson M, Skylark WJ, Freeman ALJ, Spiegelhalter DJ. Redevelopment of the predict: breast cancer website and recommendations for developing interfaces to support decision-making. Cancer Med. 2021;10(15):5141\u201353.","journal-title":"Cancer Med"},{"issue":"6","key":"16161_CR26","doi-asserted-by":"publisher","first-page":"e15154","DOI":"10.2196\/15154","volume":"22","author":"O Asan","year":"2020","unstructured":"Asan O, Bayrak AE, Choudhury A. Artificial intelligence and human trust in healthcare: focus on clinicians. J Med Internet Res. 2020;22(6):e15154.","journal-title":"J Med Internet Res"}],"container-title":["BMC Cancer"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12885-026-16161-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12885-026-16161-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12885-026-16161-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,30]],"date-time":"2026-05-30T08:31:59Z","timestamp":1780129919000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s12885-026-16161-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,28]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["16161"],"URL":"https:\/\/doi.org\/10.1186\/s12885-026-16161-5","relation":{},"ISSN":["1471-2407"],"issn-type":[{"value":"1471-2407","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,28]]},"assertion":[{"value":"23 April 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This usability study was not a clinical study, and did not require ethical approval from the Swedish Ethical Review Authority. The usability study was approved by the Head of Research in Region Sk\u00e5ne (2nd Apr, 2024) regarding the intended conduct of the study, and Region Sk\u00e5ne is thus the legal authority for the study. All participants in the study provided written informed consent.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"691"}}