{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T17:26:49Z","timestamp":1770744409308,"version":"3.49.0"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T00:00:00Z","timestamp":1585180800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T00:00:00Z","timestamp":1585180800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["SCH-1622542"],"award-info":[{"award-number":["SCH-1622542"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["SCH-1622679"],"award-info":[{"award-number":["SCH-1622679"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U01 CA154601"],"award-info":[{"award-number":["U01 CA154601"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U24 CA180927"],"award-info":[{"award-number":["U24 CA180927"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["U24 CA180918"],"award-info":[{"award-number":["U24 CA180918"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["P41EB015896"],"award-info":[{"award-number":["P41EB015896"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["K12EY027720"],"award-info":[{"award-number":["K12EY027720"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["R01EY019474"],"award-info":[{"award-number":["R01EY019474"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["K12EY027720"],"award-info":[{"award-number":["K12EY027720"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"U.S. Department of Health & Human Services | National Institutes of Health","doi-asserted-by":"publisher","award":["P30EY10572"],"award-info":[{"award-number":["P30EY10572"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"DOI":"10.13039\/100000070","name":"U.S. Department of Health & Human Services | NIH | National Institute of Biomedical Imaging and Bioengineering","doi-asserted-by":"publisher","award":["5T32EB1680"],"award-info":[{"award-number":["5T32EB1680"]}],"id":[{"id":"10.13039\/100000070","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"U.S. Department of Health & Human Services | NIH | National Cancer Institute","doi-asserted-by":"publisher","award":["F30CA239407"],"award-info":[{"award-number":["F30CA239407"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"},{"name":"U.S. Department of Health & Human Services | National Institutes of Health"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Using medical images to evaluate disease severity and change over time is a routine and important task in clinical decision making. Grading systems are often used, but are unreliable as domain experts disagree on disease severity category thresholds. These discrete categories also do not reflect the underlying continuous spectrum of disease severity. To address these issues, we developed a convolutional Siamese neural network approach to evaluate disease severity at single time points and change between longitudinal patient visits on a continuous spectrum. We demonstrate this in two medical imaging domains: retinopathy of prematurity (ROP) in retinal photographs and osteoarthritis in knee radiographs. Our patient cohorts consist of 4861 images from 870 patients in the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) cohort study and 10,012 images from 3021 patients in the Multicenter Osteoarthritis Study (MOST), both of which feature longitudinal imaging data. Multiple expert clinician raters ranked 100 retinal images and 100 knee radiographs from excluded test sets for severity of ROP and osteoarthritis, respectively. The Siamese neural network output for each image in comparison to a pool of normal reference images correlates with disease severity rank (<jats:italic>\u03c1<\/jats:italic>\u2009=\u20090.87 for ROP and <jats:italic>\u03c1<\/jats:italic>\u2009=\u20090.89 for osteoarthritis), both within and between the clinical grading categories. Thus, this output can represent the continuous spectrum of disease severity at any single time point. The difference in these outputs can be used to show change over time. Alternatively, paired images from the same patient at two time points can be directly compared using the Siamese neural network, resulting in an additional continuous measure of change between images. Importantly, our approach does not require manual localization of the pathology of interest and requires only a binary label for training (same versus different). The location of disease and site of change detected by the algorithm can be visualized using an occlusion sensitivity map-based approach. For a longitudinal binary change detection task, our Siamese neural networks achieve test set receiving operator characteristic area under the curves (AUCs) of up to 0.90 in evaluating ROP or knee osteoarthritis change, depending on the change detection strategy. The overall performance on this binary task is similar compared to a conventional convolutional deep-neural network trained for multi-class classification. Our results demonstrate that convolutional Siamese neural networks can be a powerful tool for evaluating the continuous spectrum of disease severity and change in medical imaging.<\/jats:p>","DOI":"10.1038\/s41746-020-0255-1","type":"journal-article","created":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T11:03:00Z","timestamp":1585220580000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":111,"title":["Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6207-4940","authenticated-orcid":false,"given":"Matthew D.","family":"Li","sequence":"first","affiliation":[]},{"given":"Ken","family":"Chang","sequence":"additional","affiliation":[]},{"given":"Ben","family":"Bearce","sequence":"additional","affiliation":[]},{"given":"Connie Y.","family":"Chang","sequence":"additional","affiliation":[]},{"given":"Ambrose J.","family":"Huang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7964-9475","authenticated-orcid":false,"given":"J. Peter","family":"Campbell","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7636-4554","authenticated-orcid":false,"given":"James M.","family":"Brown","sequence":"additional","affiliation":[]},{"given":"Praveer","family":"Singh","sequence":"additional","affiliation":[]},{"given":"Katharina V.","family":"Hoebel","sequence":"additional","affiliation":[]},{"given":"Deniz","family":"Erdo\u011fmu\u015f","sequence":"additional","affiliation":[]},{"given":"Stratis","family":"Ioannidis","sequence":"additional","affiliation":[]},{"given":"William E.","family":"Palmer","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8172-7636","authenticated-orcid":false,"given":"Michael F.","family":"Chiang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8906-9618","authenticated-orcid":false,"given":"Jayashree","family":"Kalpathy-Cramer","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,3,26]]},"reference":[{"key":"255_CR1","doi-asserted-by":"publisher","unstructured":"Gupta, K. et al. A quantitative severity scale for retinopathy of prematurity using deep learning to monitor disease regression after treatment. JAMA Ophthalmol. (2019) https:\/\/doi.org\/10.1001\/jamaophthalmol.2019.2442.","DOI":"10.1001\/jamaophthalmol.2019.2442"},{"key":"255_CR2","doi-asserted-by":"publisher","unstructured":"Taylor, S. et al. Monitoring disease progression with a quantitative severity scale for retinopathy of prematurity using deep learning. JAMA Ophthalmol. (2019) https:\/\/doi.org\/10.1001\/jamaophthalmol.2019.2433.","DOI":"10.1001\/jamaophthalmol.2019.2433"},{"key":"255_CR3","doi-asserted-by":"publisher","first-page":"1222","DOI":"10.1016\/j.jacr.2018.05.037","volume":"15","author":"AB Rosenkrantz","year":"2018","unstructured":"Rosenkrantz, A. B., Duszak, R., Babb, J. S., Glover, M. & Kang, S. K. Discrepancy rates and clinical impact of imaging secondary interpretations: a systematic review and meta-analysis. J. Am. Coll. Radio. 15, 1222\u20131231 (2018).","journal-title":"J. Am. Coll. Radio."},{"key":"255_CR4","doi-asserted-by":"publisher","first-page":"2338","DOI":"10.1016\/j.ophtha.2016.07.026","volume":"123","author":"JP Campbell","year":"2016","unstructured":"Campbell, J. P. et al. Plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability. Ophthalmology 123, 2338\u20132344 (2016).","journal-title":"Ophthalmology"},{"key":"255_CR5","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115\u2013118 (2017).","journal-title":"Nature"},{"key":"255_CR6","doi-asserted-by":"publisher","first-page":"2402","DOI":"10.1001\/jama.2016.17216","volume":"316","author":"V Gulshan","year":"2016","unstructured":"Gulshan, V. et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316, 2402 (2016).","journal-title":"JAMA"},{"key":"255_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2018.01.034","author":"J Krause","year":"2018","unstructured":"Krause, J. et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology https:\/\/doi.org\/10.1016\/j.ophtha.2018.01.034 (2018).","journal-title":"Ophthalmology"},{"key":"255_CR8","unstructured":"Su\u00e1rez, J. L., Garc\u00eda, S. & Herrera, F. A tutorial on distance metric learning: mathematical foundations, algorithms and software. Preprint at https:\/\/arxiv.org\/abs\/1812.05944(2018)."},{"key":"255_CR9","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1142\/S0218001493000339","volume":"07","author":"J Bromley","year":"1993","unstructured":"Bromley, J. et al. Signature verification using a \u201csiamese\u201d time delay neural network. Int. J. Pattern Recognit. Artif. Intell. 07, 669\u2013688 (1993).","journal-title":"Int. J. Pattern Recognit. Artif. Intell."},{"key":"255_CR10","doi-asserted-by":"crossref","unstructured":"Zhao, F., Xu, J. & Lin, Y. Similarity measure for patients via a Siamese CNN network. International Conference on Algorithms and Architectures for Parallel Processing. 319\u2013328 (2018).","DOI":"10.1007\/978-3-030-05054-2_25"},{"key":"255_CR11","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1006376","volume":"14","author":"N Bhagwat","year":"2018","unstructured":"Bhagwat, N., Viviano, J. D., Voineskos, A. N., Chakravarty, M. M. & Initiative, A. D. N. Modeling and prediction of clinical symptom trajectories in Alzheimer\u2019s disease using longitudinal data. PLOS Comput. Biol. 14, e1006376 (2018).","journal-title":"PLOS Comput. Biol."},{"key":"255_CR12","doi-asserted-by":"crossref","unstructured":"Hadsell, R., Chopra, S. & LeCun, Y. Dimensionality reduction by learning an invariant mapping. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Vol. 2, 1735\u20131742 (2006).","DOI":"10.1109\/CVPR.2006.100"},{"key":"255_CR13","doi-asserted-by":"publisher","first-page":"803","DOI":"10.1001\/jamaophthalmol.2018.1934","volume":"136","author":"JM Brown","year":"2018","unstructured":"Brown, J. M. et al. Automated diagnosis of plus disease in retinopathy of prematurity using deep convolutional neural networks. JAMA Ophthalmol. 136, 803\u2013810 (2018).","journal-title":"JAMA Ophthalmol."},{"key":"255_CR14","doi-asserted-by":"publisher","first-page":"1684","DOI":"10.1001\/archopht.121.12.1684","volume":"121","author":"Early Treatment For Retinopathy Of Prematurity Cooperative Group.","year":"2003","unstructured":"Early Treatment For Retinopathy Of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity. Arch. Ophthalmol. 121, 1684 (2003).","journal-title":"Arch. Ophthalmol."},{"key":"255_CR15","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1001\/archophthalmol.2011.63","volume":"129","author":"DK Wallace","year":"2011","unstructured":"Wallace, D. K., Freedman, S. F., Hartnett, M. E. & Quinn, G. E. Predictive value of pre-plus disease in retinopathy of prematurity. Arch. Ophthalmol. (Chic., Ill. 1960) 129, 591\u2013596 (2011).","journal-title":"Arch. Ophthalmol. (Chic., Ill. 1960)"},{"key":"255_CR16","doi-asserted-by":"publisher","first-page":"1886","DOI":"10.1007\/s11999-016-4732-4","volume":"474","author":"MD Kohn","year":"2016","unstructured":"Kohn, M. D., Sassoon, A. A. & Fernando, N. D. Classifications in brief: Kellgren-Lawrence classification of osteoarthritis. Clin. Orthop. Relat. Res. 474, 1886\u20131893 (2016).","journal-title":"Clin. Orthop. Relat. Res."},{"key":"255_CR17","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-20132-7","volume":"8","author":"A Tiulpin","year":"2018","unstructured":"Tiulpin, A., Thevenot, J., Rahtu, E., Lehenkari, P. & Saarakkala, S. Automatic knee osteoarthritis diagnosis from plain radiographs: a deep learning-based approach. Sci. Rep. 8, 1727 (2018).","journal-title":"Sci. Rep."},{"key":"255_CR18","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 770\u2013778 (2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"255_CR19","doi-asserted-by":"crossref","unstructured":"Tiulpin A., Thevenot J., Rahtu E., & Saarakkala S. A Novel Method for Automatic Localization of Joint Area on Knee Plain Radiographs. In: Sharma P., Bianchi F. (eds) Image Analysis. SCIA 2017. Lecture Notes in Computer Science, vol 10270 (2017).","DOI":"10.1007\/978-3-319-59129-2_25"},{"key":"255_CR20","doi-asserted-by":"publisher","first-page":"2345","DOI":"10.1016\/j.ophtha.2016.07.020","volume":"123","author":"J Kalpathy-Cramer","year":"2016","unstructured":"Kalpathy-Cramer, J. et al. Plus disease in retinopathy of prematurity: improving diagnosis by ranking disease severity and using quantitative image analysis. Ophthalmology 123, 2345\u20132351 (2016).","journal-title":"Ophthalmology"},{"key":"255_CR21","doi-asserted-by":"publisher","unstructured":"Klara, K. et al. Reliability and accuracy of cross-sectional radiographic assessment of severe knee osteoarthritis: Role of training and experience. J. Rheumatol. https:\/\/doi.org\/10.3899\/jrheum.151300 (2016).","DOI":"10.3899\/jrheum.151300"},{"key":"255_CR22","doi-asserted-by":"publisher","unstructured":"Gschlie\u00dfer, A. et al. Inter-expert and intra-expert agreement on the diagnosis and treatment of retinopathy of prematurity. Am. J. Ophthalmol. https:\/\/doi.org\/10.1016\/j.ajo.2015.05.016 (2015).","DOI":"10.1016\/j.ajo.2015.05.016"},{"key":"255_CR23","doi-asserted-by":"crossref","unstructured":"Antony, J., McGuinness, K., Connor, N. E. O. & Moran, K. Quantifying radiographic knee osteoarthritis severity using deep convolutional neural networks. Preprint at https:\/\/arxiv.org\/abs\/1609.02469 (2016).","DOI":"10.1109\/ICPR.2016.7899799"},{"key":"255_CR24","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1016\/j.neunet.2019.06.004","volume":"118","author":"I\u0307 Y\u0131ld\u0131z","year":"2019","unstructured":"Y\u0131ld\u0131z, I\u0307. et al. Classification and comparison via neural networks. Neural Netw. 118, 65\u201380 (2019).","journal-title":"Neural Netw."},{"key":"255_CR25","doi-asserted-by":"publisher","unstructured":"Schroff, F., Kalenichenko, D. & Philbin, J. FaceNet: a unified embedding for face recognition and clustering. in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition https:\/\/doi.org\/10.1109\/CVPR.2015.7298682 (2015).","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"255_CR26","first-page":"1902","volume":"2014","author":"MC Ryan","year":"2014","unstructured":"Ryan, M. C. et al. Development and evaluation of reference standards for image-based telemedicine diagnosis and clinical research studies in ophthalmology. AMIA Annu. Symp. Proc. AMIA Symp. 2014, 1902\u20131910 (2014).","journal-title":"AMIA Annu. Symp. Proc. AMIA Symp."},{"key":"255_CR27","doi-asserted-by":"crossref","unstructured":"He, K., Girshick, R. & Doll\u00e1r, P. Rethinking ImageNet pre-training. IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South) pp. 4917\u20134926 (2019).","DOI":"10.1109\/ICCV.2019.00502"},{"key":"255_CR28","unstructured":"Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https:\/\/arxiv.org\/abs\/1412.6980 (2014)."},{"key":"255_CR29","doi-asserted-by":"crossref","unstructured":"Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In: Fleet D., Pajdla T., Schiele B., Tuytelaars T. (eds) Computer Vision \u2013 ECCV 2014. Lecture Notes in Computer Science, vol 8689. Springer, Cham (2014).","DOI":"10.1007\/978-3-319-10590-1_53"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0255-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0255-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0255-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T02:07:52Z","timestamp":1670378872000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-020-0255-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,26]]},"references-count":29,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["255"],"URL":"https:\/\/doi.org\/10.1038\/s41746-020-0255-1","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,26]]},"assertion":[{"value":"21 July 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 March 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The M.F.C is an unpaid member of the scientific advisory board for Clarity Medical Systems (Pleasanton, CA), a consultant for Novartis (Basel, Switzerland), and an initial member of Inteleretina, LLC (Honolulu, HI). J.P.C. and M.F.C. receive research support from Genentech. J.K. is a consultant\/advisory board member for Infotech, Soft. The other authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"48"}}