{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T14:31:34Z","timestamp":1780410694823,"version":"3.54.1"},"reference-count":16,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T00:00:00Z","timestamp":1671753600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T00:00:00Z","timestamp":1671753600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100005625","name":"RSNA Research and Education Foundation","doi-asserted-by":"publisher","award":["2020 Canon Medical Systems USA"],"award-info":[{"award-number":["2020 Canon Medical Systems USA"]}],"id":[{"id":"10.13039\/100005625","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005625","name":"RSNA Research and Education Foundation","doi-asserted-by":"publisher","award":["Inc.\/RSNA Research Seed Grant #RSD2027"],"award-info":[{"award-number":["Inc.\/RSNA Research Seed Grant #RSD2027"]}],"id":[{"id":"10.13039\/100005625","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007052","name":"Memorial Sloan-Kettering Cancer Center","doi-asserted-by":"publisher","award":["Radiology AI Grant (internal award)"],"award-info":[{"award-number":["Radiology AI Grant (internal award)"]}],"id":[{"id":"10.13039\/100007052","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100003193","name":"Foundation of the American Society of Neuroradiology","doi-asserted-by":"publisher","award":["AI Grant"],"award-info":[{"award-number":["AI Grant"]}],"id":[{"id":"10.13039\/100003193","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Imaging"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Supervised deep learning in radiology suffers from notorious inherent limitations: 1) It requires large, hand-annotated data sets; (2) It is non-generalizable; and (3) It lacks explainability and intuition. It has recently been proposed that reinforcement learning addresses all three of these limitations. Notable prior work applied deep reinforcement learning to localize brain tumors with radiologist eye tracking points, which limits the state-action space. Here, we generalize Deep Q Learning to a gridworld-based environment so that only the images and image masks are required.\n<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We trained a Deep <jats:inline-formula><jats:alternatives><jats:tex-math>$$Q$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mi>Q<\/mml:mi>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> network on 30 two-dimensional image slices from the BraTS brain tumor database. Each image contained one lesion. We then tested the trained Deep Q network on a separate set of 30 testing set images. For comparison, we also trained and tested a keypoint detection supervised deep learning network on the same set of training\/testing images.\n<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Whereas the supervised approach quickly overfit the training data and predictably performed poorly on the testing set (11% accuracy), the Deep <jats:inline-formula><jats:alternatives><jats:tex-math>$$Q$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                    <mml:mi>Q<\/mml:mi>\n                  <\/mml:math><\/jats:alternatives><\/jats:inline-formula> learning approach showed progressive improved generalizability to the testing set over training time, reaching 70% accuracy.\n<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>We have successfully applied reinforcement learning to localize brain tumors on 2D contrast-enhanced MRI brain images. This represents a generalization of recent work to a gridworld setting naturally suitable for analyzing medical images. We have shown that reinforcement learning does not over-fit small training sets, and can generalize to a separate testing set.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12880-022-00919-x","type":"journal-article","created":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T09:02:37Z","timestamp":1671786157000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Reinforcement learning using Deep $$Q$$ networks and $$Q$$ learning accurately localizes brain tumors on MRI with very small training sets"],"prefix":"10.1186","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3169-9590","authenticated-orcid":false,"given":"J. N.","family":"Stember","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"H.","family":"Shalu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,12,23]]},"reference":[{"key":"919_CR1","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1016\/j.media.2019.02.007","volume":"53","author":"A Alansary","year":"2019","unstructured":"Alansary A, et al. Evaluating reinforcement learning agents for anatomical landmark detection. Med Image Anal. 2019;53:156\u201364.","journal-title":"Med Image Anal"},{"key":"919_CR2","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1109\/TPAMI.2017.2782687","volume":"41","author":"F-C Ghesu","year":"2017","unstructured":"Ghesu F-C, et al. Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans. IEEE Trans Pattern Anal Mach Intell. 2017;41:176\u201389.","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"919_CR3","doi-asserted-by":"crossref","unstructured":"Zhou SK, Le HN, Luu K, Nguyen HV, Ayache N. Deep reinforcement learning in medical imaging: a literature review. 2021; arXiv preprint, arXiv:2103.05115.","DOI":"10.1016\/j.media.2021.102193"},{"key":"919_CR4","first-page":"1245","volume":"39","author":"WA Al","year":"2019","unstructured":"Al WA, Yun ID. Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images. IEEE Trans Med Imaging. 2019;39:1245\u201355.","journal-title":"IEEE Trans Med Imaging"},{"key":"919_CR5","doi-asserted-by":"crossref","unstructured":"Maicas G, Carneiro G, Bradley AP, Nascimento JC, Reid I. Deep reinforcement learning for active breast lesion detection from DCE-MRI. In: International Conference on Medical Image Computing and Computer Assisted Intervention. 2017; 665\u2013673.","DOI":"10.1007\/978-3-319-66179-7_76"},{"key":"919_CR6","doi-asserted-by":"publisher","first-page":"108","DOI":"10.3389\/fonc.2018.00108","volume":"8","author":"I Ali","year":"2018","unstructured":"Ali I, et al. Lung nodule detection via deep reinforcement learning. Front Oncol. 2018;8:108.","journal-title":"Front Oncol"},{"key":"919_CR7","doi-asserted-by":"publisher","first-page":"6187","DOI":"10.3390\/s21186187","volume":"21","author":"Y Jang","year":"2021","unstructured":"Jang Y, Jeon B. Deep reinforcement learning with explicit spatiosequential encoding network for coronary ostia identification in CT images. Sensors. 2021;21:6187.","journal-title":"Sensors"},{"key":"919_CR8","doi-asserted-by":"crossref","unstructured":"Zhang P, Wang F, Zheng Y. Deep reinforcement learning for vessel centerline tracing in multi-modality 3D volumes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. 2018, 755\u2013763.","DOI":"10.1007\/978-3-030-00937-3_86"},{"key":"919_CR9","unstructured":"Stember J, Shalu H. Deep reinforcement learning to detect brain lesions on MRI: a proof-of-concept application of reinforcement learning to medical images. 2020; arXiv preprint arXiv:2008.02708."},{"key":"919_CR10","doi-asserted-by":"crossref","unstructured":"Wang X et al. Inconsistent Performance of deep learning models on mammogram classification. J Am Coll Radiol; 2020.","DOI":"10.1016\/j.jacr.2020.01.006"},{"key":"919_CR11","unstructured":"Goodfellow IJ, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. 2014; arXiv preprint arXiv:1412.6572."},{"key":"919_CR12","unstructured":"Buhrmester V, M\u00fcnch D, Arens M. Analysis of explainers of black box deep neural networks for computer vision: a survey; 2019. arXiv preprint arXiv:1911.12116"},{"key":"919_CR13","doi-asserted-by":"publisher","first-page":"e271","DOI":"10.1016\/S2589-7500(19)30123-2","volume":"1","author":"X Liu","year":"2019","unstructured":"Liu X, et al. A comparison of deep learning performance against healthcare professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. 2019;1:e271\u201397.","journal-title":"Lancet Digit Health"},{"key":"919_CR14","doi-asserted-by":"publisher","first-page":"1993","DOI":"10.1109\/TMI.2014.2377694","volume":"34","author":"BH Menze","year":"2014","unstructured":"Menze BH, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging. 2014;34:1993\u20132024.","journal-title":"IEEE Trans Med Imaging"},{"key":"919_CR15","unstructured":"Sutton RS, Barto AG. Reinforcement learning: an introduction, MIT press; 2018."},{"key":"919_CR16","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2020200047","author":"JN Stember","year":"2021","unstructured":"Stember JN, et al. Integrating eye tracking and speech recognition accurately annotates MR brain images for deep learning: proof of principle. Radiol Artif Intell. 2021. https:\/\/doi.org\/10.1148\/ryai.2020200047.","journal-title":"Radiol Artif Intell"}],"container-title":["BMC Medical Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12880-022-00919-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12880-022-00919-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12880-022-00919-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,23]],"date-time":"2022-12-23T09:03:50Z","timestamp":1671786230000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedimaging.biomedcentral.com\/articles\/10.1186\/s12880-022-00919-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,23]]},"references-count":16,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["919"],"URL":"https:\/\/doi.org\/10.1186\/s12880-022-00919-x","relation":{},"ISSN":["1471-2342"],"issn-type":[{"value":"1471-2342","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,23]]},"assertion":[{"value":"12 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 October 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable given no protected health information used.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"We are seeking a patent based in part on the methods detailed here. The patent does not restrict research applications of the method or work presented in this paper. We have recently launched a radiology AI startup, Authera Inc, that will probably apply some of the methods described here.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interest"}}],"article-number":"224"}}