{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T20:30:56Z","timestamp":1774989056294,"version":"3.50.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2021,12,15]],"date-time":"2021-12-15T00:00:00Z","timestamp":1639526400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,12,15]],"date-time":"2021-12-15T00:00:00Z","timestamp":1639526400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Segmenting subcellular structures in living cells from fluorescence microscope images is a ground truth (GT)-deficient problem. The microscopes\u2019 three-dimensional blurring function, finite optical resolution due to light diffraction, finite pixel resolution and the complex morphological manifestations of the structures all contribute to GT-hardness. Unsupervised segmentation approaches are quite inaccurate. Therefore, manual segmentation relying on heuristics and experience remains the preferred approach. However, this process is tedious, given the countless structures present inside a single cell, and generating analytics across a large population of cells or performing advanced artificial intelligence tasks such as tracking are greatly limited. Here we bring modelling and deep learning to a nexus for solving this GT-hard problem, improving both the accuracy and speed of subcellular segmentation. We introduce a simulation-supervision approach empowered by physics-based GT, which presents two advantages. First, the physics-based GT resolves the GT-hardness. Second, computational modelling of all the relevant physical aspects assists the deep learning models in learning to compensate, to a great extent, for the limitations of physics and the instrument. We show extensive results on the segmentation of small vesicles and mitochondria in diverse and independent living- and fixed-cell datasets. We demonstrate the adaptability of the approach across diverse microscopes through transfer learning, and illustrate biologically relevant applications of automated analytics and motion analysis.<\/jats:p>","DOI":"10.1038\/s42256-021-00420-0","type":"journal-article","created":{"date-parts":[[2021,12,15]],"date-time":"2021-12-15T17:02:52Z","timestamp":1639587772000},"page":"1071-1080","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Physics-based machine learning for subcellular segmentation in living cells"],"prefix":"10.1038","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0706-2565","authenticated-orcid":false,"given":"Arif Ahmed","family":"Sekh","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4462-4600","authenticated-orcid":false,"given":"Ida S.","family":"Opstad","sequence":"additional","affiliation":[]},{"given":"Gustav","family":"Godtliebsen","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1080-3619","authenticated-orcid":false,"given":"\u212bsa Birna","family":"Birgisdottir","sequence":"additional","affiliation":[]},{"given":"Balpreet Singh","family":"Ahluwalia","sequence":"additional","affiliation":[]},{"given":"Krishna","family":"Agarwal","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3693-6973","authenticated-orcid":false,"given":"Dilip K.","family":"Prasad","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,12,15]]},"reference":[{"key":"420_CR1","doi-asserted-by":"crossref","unstructured":"Chang, S. W. & Liao, S. W. KUnet: microscopy image segmentation with deep Unet based convolutional networks. In Proc. IEEE International Conference on Systems, Man and Cybernetics 3561\u20133566 (IEEE, 2019).","DOI":"10.1109\/SMC.2019.8914048"},{"key":"420_CR2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-030-00889-5_1","volume":"11045","author":"Z Zhou","year":"2018","unstructured":"Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N. & Liang, J. UNet++: a nested U-Net architecture for medical image segmentation. Feep Learn. Med. Image Anal. Multimodal Learn. Clin. Decis. Support 11045, 3\u201311 (2018).","journal-title":"Feep Learn. Med. Image Anal. Multimodal Learn. Clin. Decis. Support"},{"key":"420_CR3","doi-asserted-by":"publisher","first-page":"e1005177","DOI":"10.1371\/journal.pcbi.1005177","volume":"12","author":"DA Van Valen","year":"2016","unstructured":"Van Valen, D. A. et al. Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol. 12, e1005177 (2016).","journal-title":"PLoS Comput. Biol."},{"key":"420_CR4","doi-asserted-by":"publisher","first-page":"7860","DOI":"10.1038\/s41598-017-07599-6","volume":"7","author":"SK Sadanandan","year":"2017","unstructured":"Sadanandan, S. K., Ranefall, P., Le Guyader, S. & W\u00e4hlby, C. Automated training of deep convolutional neural networks for cell segmentation. Sci. Rep. 7, 7860 (2017).","journal-title":"Sci. Rep."},{"key":"420_CR5","doi-asserted-by":"crossref","unstructured":"Aydin, A. S., Dubey, A., Dovrat, D., Aharoni, A. & Shilkrot, R. CNN based yeast cell segmentation in multi-modal fluorescent microscopy data. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops 753\u2013759 (IEEE, 2017).","DOI":"10.1109\/CVPRW.2017.105"},{"key":"420_CR6","doi-asserted-by":"crossref","unstructured":"Majurski, M. et al. Cell image segmentation using generative adversarial networks, transfer learning and augmentations. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops 1114\u20131122 (IEEE, 2019).","DOI":"10.1109\/CVPRW.2019.00145"},{"key":"420_CR7","doi-asserted-by":"crossref","unstructured":"Sekh, A. A. et al. Learning nanoscale motion patterns of vesicles in living cells. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 14014\u201314023 (IEEE, 2020).","DOI":"10.1109\/CVPR42600.2020.01403"},{"key":"420_CR8","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1016\/j.patrec.2011.01.021","volume":"32","author":"X Xu","year":"2011","unstructured":"Xu, X., Xu, S., Jin, L. & Song, E. Characteristic analysis of Otsu threshold and its applications. Pattern Recogn. Lett. 32, 956\u2013961 (2011).","journal-title":"Pattern Recogn. Lett."},{"key":"420_CR9","unstructured":"Arganda-Carreras, I. & Legland, D. Morphological segmentation (IJPB-plugins). ImageJ https:\/\/imagej.net\/Morphological_Segmentation (2019)."},{"key":"420_CR10","doi-asserted-by":"crossref","unstructured":"Shibuya, E. & Hotta, K. Feedback U-Net for cell image segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops 974\u2013975 (IEEE, 2020).","DOI":"10.1109\/CVPRW50498.2020.00495"},{"key":"420_CR11","doi-asserted-by":"publisher","first-page":"120059","DOI":"10.1016\/j.biomaterials.2020.120059","volume":"250","author":"Q Chen","year":"2020","unstructured":"Chen, Q. et al. Quantitative analysis of interactive behavior of mitochondria and lysosomes using structured illumination microscopy. Biomaterials 250, 120059 (2020).","journal-title":"Biomaterials"},{"key":"420_CR12","doi-asserted-by":"publisher","first-page":"13752","DOI":"10.1038\/ncomms13752","volume":"7","author":"K Agarwal","year":"2016","unstructured":"Agarwal, K. & Mach\u00e1\u0148, R. Multiple signal classification algorithm for super-resolution fluorescence microscopy. Nat. Commun. 7, 13752 (2016).","journal-title":"Nat. Commun."},{"key":"420_CR13","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (eds Navab N. et al.) 234\u2013241 (Springer, 2015).","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"420_CR14","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1523\/ENEURO.0056-18.2018","volume":"5","author":"SP Shen","year":"2018","unstructured":"Shen, S. P. et al. Automatic cell segmentation by adaptive thresholding (ACSAT) for large-scale calcium imaging datasets. eNeuro 5, 56 (2018).","journal-title":"eNeuro"},{"key":"420_CR15","doi-asserted-by":"crossref","unstructured":"Kanezaki, A. Unsupervised image segmentation by backpropagation. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing 1543\u20131547 (IEEE, 2018).","DOI":"10.1109\/ICASSP.2018.8462533"},{"key":"420_CR16","doi-asserted-by":"publisher","first-page":"2897","DOI":"10.7150\/thno.40495","volume":"10","author":"Y Li","year":"2020","unstructured":"Li, Y. et al. Imaging of macrophage mitochondria dynamics in vivo reveals cellular activation phenotype for diagnosis. Theranostics 10, 2897\u20132917 (2020).","journal-title":"Theranostics"},{"key":"420_CR17","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1016\/j.bbamcr.2014.11.002","volume":"1853","author":"AP Leonard","year":"2015","unstructured":"Leonard, A. P. et al. Quantitative analysis of mitochondrial morphology and membrane potential in living cells using high-content imaging, machine learning and morphological binning. Biochim. Biophys. Acta 1853, 348\u2013360 (2015).","journal-title":"Biochim. Biophys. Acta"},{"key":"420_CR18","doi-asserted-by":"crossref","unstructured":"Svoboda, D. & Ulman, V. Generation of synthetic image datasets for time-lapse fluorescence microscopy. In Proc. International Conference Image Analysis and Recognition (eds Campilho A. & Kamel M.) 473\u2013482 (Springer, 2012).","DOI":"10.1007\/978-3-642-31298-4_56"},{"key":"420_CR19","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/j.patcog.2019.01.006","volume":"90","author":"Z Wu","year":"2019","unstructured":"Wu, Z., Shen, C. & Van Den Hengel, A. Wider or deeper: revisiting the ResNet model for visual recognition. Pattern Recognit. 90, 119\u2013133 (2019).","journal-title":"Pattern Recognit."},{"key":"420_CR20","doi-asserted-by":"crossref","unstructured":"Qassim, H., Verma, A. & Feinzimer, D. Compressed residual-VGG16 CNN model for big data places image recognition. In Proc. IEEE Annual Computing and Communication Workshop and Conference (CCWC) 169\u2013175 (IEEE, 2018).","DOI":"10.1109\/CCWC.2018.8301729"},{"key":"420_CR21","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V. & Alemi, A. Inception-v4, inception-ResNet and the impact of residual connections on learning. In Proc. 31st AAAI Conference on Artificial Intelligence 4278\u20134284 (ACM, 2017).","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"420_CR22","doi-asserted-by":"crossref","unstructured":"Tan, M., Pang, R. & Le, Q. V. EfficientDet: scalable and efficient object detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 10781\u201310790 (IEEE, 2020).","DOI":"10.1109\/CVPR42600.2020.01079"},{"key":"420_CR23","doi-asserted-by":"publisher","first-page":"e4312","DOI":"10.7717\/peerj.4312","volume":"6","author":"J Nunez-Iglesias","year":"2018","unstructured":"Nunez-Iglesias, J., Blanch, A. J., Looker, O., Dixon, M. W. & Tilley, L. A new Python library to analyse skeleton images confirms malaria parasite remodelling of the red blood cell membrane skeleton. PeerJ 6, e4312 (2018).","journal-title":"PeerJ"},{"key":"420_CR24","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1016\/j.jvcir.2006.03.004","volume":"17","author":"S-K Weng","year":"2006","unstructured":"Weng, S.-K., Kuo, C.-M. & Tu, S.-K. Video object tracking using adaptive Kalman filter. J. Vis. Commun. Image Represent. 17, 1190\u20131208 (2006).","journal-title":"J. Vis. Commun. Image Represent."},{"key":"420_CR25","doi-asserted-by":"crossref","unstructured":"Wojke, N., Bewley, A. & Paulus, D. Simple online and realtime tracking with a deep association metric. In Proc. IEEE International Conference on Image Processing (ICIP) 3645\u20133649 (IEEE, 2017).","DOI":"10.1109\/ICIP.2017.8296962"},{"key":"420_CR26","doi-asserted-by":"publisher","unstructured":"Opstad, I. S., Godtliebsen, G. & Sekh, A. A. Replication data for: physics based machine learning for sub-cellular segmentation in living cells. DataverseNO https:\/\/doi.org\/10.18710\/IDCUCI (2021).","DOI":"10.18710\/IDCUCI"},{"key":"420_CR27","doi-asserted-by":"publisher","unstructured":"Sekh, A. A. skarifahmed\/physeg: physics based machine learning for sub-cellular segmentation in living cells. Zenodo https:\/\/doi.org\/10.5281\/zenodo.5017066 (2021).","DOI":"10.5281\/zenodo.5017066"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-021-00420-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-021-00420-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-021-00420-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T20:41:46Z","timestamp":1670100106000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-021-00420-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,15]]},"references-count":27,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["420"],"URL":"https:\/\/doi.org\/10.1038\/s42256-021-00420-0","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,15]]},"assertion":[{"value":"21 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 December 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}