{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,5]],"date-time":"2026-11-05T05:08:10Z","timestamp":1793855290255,"version":"4.6.1"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,6,21]],"date-time":"2019-06-21T00:00:00Z","timestamp":1561075200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,6,21]],"date-time":"2019-06-21T00:00:00Z","timestamp":1561075200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Google, LLC provided the funding for this research."}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Although these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. As pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep-learning-based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY\u2019s ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist\u2019s arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.<\/jats:p>","DOI":"10.1038\/s41746-019-0131-z","type":"journal-article","created":{"date-parts":[[2019,6,21]],"date-time":"2019-06-21T06:02:39Z","timestamp":1561096959000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":133,"title":["Similar image search for histopathology: SMILY"],"prefix":"10.1038","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8422-2908","authenticated-orcid":false,"given":"Narayan","family":"Hegde","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason D.","family":"Hipp","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Emmert-Buck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily","family":"Reif","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Smilkov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Terry","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carrie J.","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahul B.","family":"Amin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0816-3395","authenticated-orcid":false,"given":"Craig H.","family":"Mermel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Phil Q.","family":"Nelson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lily H.","family":"Peng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Greg S.","family":"Corrado","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin C.","family":"Stumpe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2019,6,21]]},"reference":[{"key":"131_CR1","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1097\/PAS.0000000000000948","volume":"42","author":"S Mukhopadhyay","year":"2018","unstructured":"Mukhopadhyay, S. et al. Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: a multicenter blinded randomized noninferiority study of 1992 cases (Pivotal Study). Am. J. Surg. Pathol. 42, 39\u201352 (2018).","journal-title":"Am. J. Surg. Pathol."},{"key":"131_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/1126004.1126005","volume":"2","author":"MS Lew","year":"2006","unstructured":"Lew, M. S., Sebe, N., Djeraba, C. & Jain, R. Content-based multimedia information retrieval: state of the art and challenges. ACM Trans. Multimed. Comput. Commun. Appl. 2, 1\u201319 (2006).","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"131_CR3","doi-asserted-by":"publisher","first-page":"1490","DOI":"10.5858\/arpa.2017-0222-RA","volume":"141","author":"LM Stanoszek","year":"2017","unstructured":"Stanoszek, L. M., Wang, G. Y. & Harms, P. W. Histologic mimics of basal cell carcinoma. Arch. Pathol. Lab. Med. 141, 1490\u20131502 (2017).","journal-title":"Arch. Pathol. Lab. Med."},{"key":"131_CR4","unstructured":"Google Images. https:\/\/images.google.com\/. (Accessed 17 Jan 2019)."},{"key":"131_CR5","unstructured":"Amazon Flow. A9. https:\/\/a9.com\/what-we-do\/visual-search.html. (Accessed 17 Jan 2019)."},{"key":"131_CR6","doi-asserted-by":"crossref","unstructured":"Borovikov, E. & Vajda, S. Facematch: Real-world Face Image Retrieval. In 2016 International Conference on Recent Trends in Image Processing and Pattern Recognition (RTIP2R). (Springer, Singapore, 2016).","DOI":"10.1007\/978-981-10-4859-3_36"},{"key":"131_CR7","doi-asserted-by":"publisher","first-page":"515","DOI":"10.1007\/978-3-642-24469-8_62","volume-title":"Research and Advanced Technology for Digital Libraries","author":"Krassimira Ivanova","year":"2011","unstructured":"Ivanova, K. Content-Based Image Retrieval in Digital Libraries of Art Images Utilizing Colour Semantics. (eds Gradmann, S., Borri, F., Meghini, C., Schuldt, H.) 2011 Research and Advanced Technology for Digital Libraries (TPDL). Lecture Notes in Computer Science, Vol. 6966. (Springer, Berlin, Heidelberg, 2011)."},{"key":"131_CR8","doi-asserted-by":"crossref","unstructured":"Sklan, J. E. S., Plassard, A. J., Fabbri, D. & Landman, B. A. Toward content based image retrieval with deep convolutional neural networks. (eds Gimi, B., Molthen, R. C.) Proc. SPIE\u2014the International Society for Optical Engineering. Vol. 9417 (SPIE, Bellingham, WA, 2015).","DOI":"10.1117\/12.2081551"},{"key":"131_CR9","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0181707","volume":"12","author":"J Ahmad","year":"2017","unstructured":"Ahmad, J., Sajjad, M., Mehmood, I. & Baik, S. W. SiNC: Saliency-injected neural codes for representation and efficient retrieval of medical radiographs. PLoS ONE 12, e0181707 (2017).","journal-title":"PLoS ONE"},{"key":"131_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ijmedinf.2003.11.024","volume":"73","author":"H M\u00fcller","year":"2004","unstructured":"M\u00fcller, H., Michoux, N., Bandon, D. & Geissbuhler, A. A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int. J. Med. Inform. 73, 1\u201323 (2004).","journal-title":"Int. J. Med. Inform."},{"key":"131_CR11","unstructured":"Wang, J. Z. Pathfinder: multiresolution region-based searching of pathology images using IRM. (ed. Overhage, J.M.) Proc. AMIA Symp. 883\u2013887 (Hanley & Belfus, Inc, Philadelphia, PA, 2000)."},{"key":"131_CR12","doi-asserted-by":"publisher","unstructured":"Komura, D. et al. Luigi: Large-scale histopathological image retrieval system using deep texture representations. bioRxiv 345785 (2018). https:\/\/doi.org\/10.1101\/345785","DOI":"10.1101\/345785"},{"key":"131_CR13","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2105-15-287","volume":"15","author":"X Qi","year":"2014","unstructured":"Qi, X. et al. Content-based histopathology image retrieval using CometCloud. BMC Bioinforma. 15, 287 (2014).","journal-title":"BMC Bioinforma."},{"key":"131_CR14","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.1109\/TMI.2018.2796130","volume":"37","author":"Y Zheng","year":"2018","unstructured":"Zheng, Y. et al. Histopathological whole slide image analysis using context-based CBIR. IEEE Trans. Med. Imaging 37, 1641\u20131652 (2018).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"131_CR15","doi-asserted-by":"publisher","first-page":"41","DOI":"10.4103\/2153-3539.159441","volume":"6","author":"A Sridhar","year":"2015","unstructured":"Sridhar, A., Doyle, S. & Madabhushi, A. Content-based image retrieval of digitized histopathology in boosted spectrally embedded spaces. J. Pathol. Inform. 6, 41 (2015).","journal-title":"J. Pathol. Inform."},{"key":"131_CR16","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1109\/RBME.2014.2340401","volume":"8","author":"C Mosquera-Lopez","year":"2015","unstructured":"Mosquera-Lopez, C., Agaian, S., Velez-Hoyos, A. & Thompson, I. Computer-aided prostate cancer diagnosis from digitized histopathology: a review on texture-based systems. IEEE Rev. Biomed. Eng. 8, 98\u2013113 (2015).","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"131_CR17","first-page":"3719","volume":"2009","author":"N Mehta","year":"2009","unstructured":"Mehta, N., Alomari, R. S. & Chaudhary, V. Content based sub-image retrieval system for high resolution pathology images using salient interest points. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2009, 3719\u20133722 (2009).","journal-title":"Conf. Proc. IEEE Eng. Med. Biol. Soc."},{"key":"131_CR18","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-016-1086-6","volume":"17","author":"JT Kwak","year":"2016","unstructured":"Kwak, J. T., Hewitt, S. M., Kajdacsy-Balla, A. A., Sinha, S. & Bhargava, R. Automated prostate tissue referencing for cancer detection and diagnosis. BMC Bioinforma. 17, 227 (2016).","journal-title":"BMC Bioinforma."},{"key":"131_CR19","doi-asserted-by":"crossref","unstructured":"Babaie, M. et al. Classification and retrieval of digital pathology scans: a new dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR). (Honolulu, HI, 2017).","DOI":"10.1109\/CVPRW.2017.106"},{"key":"131_CR20","doi-asserted-by":"publisher","unstructured":"Otalora, S., Schaer, R., Atzori, M., del Toro, O. A. J. & Muller, H. Deep learning based retrieval system for gigapixel histopathology cases and open access literature. bioRxiv 408237 (2018). https:\/\/doi.org\/10.1101\/408237","DOI":"10.1101\/408237"},{"key":"131_CR21","unstructured":"The Cancer Genome Atlas Home Page. The Cancer Genome Atlas-National Cancer Institute (2011). https:\/\/cancergenome.nih.gov\/. (Accessed 13 Dec 2018)."},{"key":"131_CR22","first-page":"2579","volume":"9","author":"L Maaten","year":"2008","unstructured":"Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579\u20132605 (2008).","journal-title":"J. Mach. Learn. Res."},{"key":"131_CR23","doi-asserted-by":"crossref","unstructured":"Jegou, H., Douze, M. & Schmid, C. Hamming embedding and weak geometric consistency for large scale image search. In Proc. 10th European Conference on Computer Vision. Part I 304\u2013317 (Springer, Berlin, Beidelberg, 2008).","DOI":"10.1007\/978-3-540-88682-2_24"},{"key":"131_CR24","doi-asserted-by":"crossref","unstructured":"Cai, C. J. et al. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proc. 2019 CHI Conference on Human Factors in Computing Systems. (ACM, New York, NY, 2019).","DOI":"10.1145\/3290605.3300234"},{"key":"131_CR25","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1111\/j.1365-2559.1991.tb00229.x","volume":"19","author":"CW Elston","year":"1991","unstructured":"Elston, C. W. & Ellis, I. O. Pathological prognostic factors in breast cancer. I. The value of histological grade in breast cancer: experience from a large study with long-term follow-up. Histopathology 19, 403\u2013410 (1991).","journal-title":"Histopathology"},{"issue":"1","key":"131_CR26","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.media.2014.11.010","volume":"20","author":"Mitko Veta","year":"2015","unstructured":"Veta, M. et al. Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med. Image. Anal. 20, 237\u2013248 (2015)","journal-title":"Medical Image Analysis"},{"key":"131_CR27","doi-asserted-by":"crossref","unstructured":"Veta, M. et al. Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge. Med. Image. Anal. 54, 111\u2013121 (2019).","DOI":"10.1016\/j.media.2019.02.012"},{"key":"131_CR28","doi-asserted-by":"crossref","unstructured":"Litjens, G. et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. Gigascience 7, giy065. (2018).","DOI":"10.1093\/gigascience\/giy065"},{"key":"131_CR29","unstructured":"Campanella, G., Silva, V. W. K. & Fuchs, T. J. Terabyte-scale deep multiple instance learning for classification and localization in pathology. Preprint at arXiv [cs.CV]. https:\/\/arxiv.org\/abs\/1805.06983 (2018)."},{"key":"131_CR30","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1007\/s10278-010-9290-9","volume":"24","author":"CB Akg\u00fcl","year":"2011","unstructured":"Akg\u00fcl, C. B. et al. Content-based image retrieval in radiology: current status and future directions. J. Digit. Imaging 24, 208\u2013222 (2011).","journal-title":"J. Digit. Imaging"},{"key":"131_CR31","doi-asserted-by":"publisher","DOI":"10.1038\/srep27306","volume":"6","author":"R Sparks","year":"2016","unstructured":"Sparks, R. & Madabhushi, A. Out-of-sample extrapolation utilizing semi-supervised manifold learning (OSE-SSL): content based image retrieval for histopathology images. Sci. Rep. 6, 27306 (2016).","journal-title":"Sci. Rep."},{"key":"131_CR32","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1016\/j.jbi.2011.01.011","volume":"44","author":"JC Caicedo","year":"2011","unstructured":"Caicedo, J. C., Gonz\u00e1lez, F. A. & Romero, E. Content-based histopathology image retrieval using a kernel-based semantic annotation framework. J. Biomed. Inform. 44, 519\u2013528 (2011).","journal-title":"J. Biomed. Inform."},{"key":"131_CR33","unstructured":"The Cancer Genome Atlas Home Page. The Cancer Genome Atlas-National Cancer Institute (2011). https:\/\/cancergenome.nih.gov\/. (Accessed 13 Dec 2018)."},{"key":"131_CR34","doi-asserted-by":"crossref","unstructured":"Wang, J. et al. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1386\u20131393 (2014).","DOI":"10.1109\/CVPR.2014.180"},{"key":"131_CR35","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"131_CR36","unstructured":"Image Similarity Data. https:\/\/sites.google.com\/site\/imagesimilaritydata. (Accessed 17 Jan 2019)."},{"key":"131_CR37","doi-asserted-by":"publisher","unstructured":"Michael Ando, D., McLean, C. Y. & Berndl, M. Improving phenotypic measurements in high-content imaging screens. bioRxiv 161422 (2017). https:\/\/doi.org\/10.1101\/161422","DOI":"10.1101\/161422"},{"key":"131_CR38","doi-asserted-by":"publisher","first-page":"209","DOI":"10.1145\/355744.355745","volume":"3","author":"JH Friedman","year":"1977","unstructured":"Friedman, J. H., Bentley, J. L. & Finkel, R. A. An algorithm for finding best matches in logarithmic expected time. ACM Trans. Math. Softw. 3, 209\u2013226 (1977).","journal-title":"ACM Trans. Math. Softw."},{"key":"131_CR39","unstructured":"NDP. view2 Viewing software U12388-01. https:\/\/www.hamamatsu.com\/jp\/en\/product\/type\/U12388-01\/index.html. (Accessed 13 Dec 2018)."},{"key":"131_CR40","unstructured":"GDC. https:\/\/portal.gdc.cancer.gov\/. (Accessed 1 Feb 2019)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0131-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0131-z","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0131-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,17]],"date-time":"2022-12-17T13:29:36Z","timestamp":1671283776000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-019-0131-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,6,21]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["131"],"URL":"https:\/\/doi.org\/10.1038\/s41746-019-0131-z","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,6,21]]},"assertion":[{"value":"31 January 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 May 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 June 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"N.H., J.D.H., Y.L., E.R., D.S., M.T., C.J.C., C.H.M., P.Q.N., L.H.P., G.S.C. and M.C.S. are employees of Google LLC and own Alphabet stock. M.E.B. and M.B.A. were compensated for their expertize and time as pathologists.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"56"}}