{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,10]],"date-time":"2025-12-10T08:57:32Z","timestamp":1765357052717,"version":"build-2065373602"},"reference-count":59,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2021,12,29]],"date-time":"2021-12-29T00:00:00Z","timestamp":1640736000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Frequently, neural network training involving biological images suffers from a lack of data, resulting in inefficient network learning. This issue stems from limitations in terms of time, resources, and difficulty in cellular experimentation and data collection. For example, when performing experimental analysis, it may be necessary for the researcher to use most of their data for testing, as opposed to model training. Therefore, the goal of this paper is to perform dataset augmentation using generative adversarial networks (GAN) to increase the classification accuracy of deep convolutional neural networks (CNN) trained on induced pluripotent stem cell microscopy images. The main challenges are: 1. modeling complex data using GAN and 2. training neural networks on augmented datasets that contain generated data. To address these challenges, a temporally constrained, hierarchical classification scheme that exploits domain knowledge is employed for model learning. First, image patches of cell colonies from gray-scale microscopy images are generated using GAN, and then these images are added to the real dataset and used to address class imbalances at multiple stages of training. Overall, a 2% increase in both true positive rate and F1-score is observed using this method as compared to a straightforward, imbalanced classification network, with some greater improvements on a classwise basis. This work demonstrates that synergistic model design involving domain knowledge is key for biological image analysis and improves model learning in high-throughput scenarios.<\/jats:p>","DOI":"10.3390\/s22010206","type":"journal-article","created":{"date-parts":[[2021,12,29]],"date-time":"2021-12-29T02:31:27Z","timestamp":1640745087000},"page":"206","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Generative Adversarial Networks for Morphological\u2013Temporal Classification of Stem Cell Images"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1097-651X","authenticated-orcid":false,"given":"Adam","family":"Witmer","sequence":"first","affiliation":[{"name":"Visualization and Intelligent Systems Laboratory, University of California, Riverside, CA 92521, USA"},{"name":"Department of Bioengineering, University of California, Riverside, CA 92521, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8971-6416","authenticated-orcid":false,"given":"Bir","family":"Bhanu","sequence":"additional","affiliation":[{"name":"Visualization and Intelligent Systems Laboratory, University of California, Riverside, CA 92521, USA"},{"name":"Department of Bioengineering, University of California, Riverside, CA 92521, USA"},{"name":"Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1126\/science.282.5391.1145","article-title":"Embryonic stem cell lines derived from human blastocysts","volume":"282","author":"Thomson","year":"1998","journal-title":"Science"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1586","DOI":"10.1016\/j.cell.2016.05.082","article-title":"Modeling development and disease with organoids","volume":"165","author":"Clevers","year":"2016","journal-title":"Cell"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1433","DOI":"10.1126\/science.287.5457.1433","article-title":"Mammalian neural stem cells","volume":"287","author":"Gage","year":"2000","journal-title":"Science"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1126\/science.284.5411.143","article-title":"Multilineage potential of adult human mesenchymal stem cells","volume":"284","author":"Pittenger","year":"1999","journal-title":"Science"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"571","DOI":"10.1038\/s41580-020-0259-3","article-title":"Human organoids: Model systems for human biology and medicine","volume":"21","author":"Kim","year":"2020","journal-title":"Nat. Rev. Mol. Cell Biol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.cell.2007.11.019","article-title":"Induction of pluripotent stem cells from adult human fibroblasts by defined factors","volume":"131","author":"Takahashi","year":"2007","journal-title":"Cell"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.cels.2017.08.009","article-title":"Stem cell differentiation as a non-Markov stochastic process","volume":"5","author":"Stumpf","year":"2017","journal-title":"Cell Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1016\/j.stem.2020.09.014","article-title":"Pluripotent stem cell-based cell therapy\u2014promise and challenges","volume":"27","author":"Yamanaka","year":"2020","journal-title":"Cell Stem Cell"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Talbot, P., Zur Nieden, N., Lin, S., Martinez, I., Guan, B., and Bhanu, B. (2014). Use of video bioinformatics tools in stem cell toxicology. Handbook of Nanotoxicology, Nanomedicine and Stem Cell Use in Toxicology, John Wiley and Sons, Ltd.","DOI":"10.1002\/9781118856017.ch21"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bhanu, B., and Talbot, P. (2015). Video Bioinformatics: From Live Imaging to Knowledge, Springer International Publishing. [1st ed.].","DOI":"10.1007\/978-3-319-23724-4"},{"key":"ref_11","unstructured":"(2021, September 21). Available online: https:\/\/www.nikon.com\/products\/microscope-solutions\/special\/ct\/."},{"key":"ref_12","first-page":"36","article-title":"Image processing with ImageJ","volume":"11","author":"Ram","year":"2004","journal-title":"Biophotonics Int."},{"key":"ref_13","unstructured":"(2021, October 05). Available online: https:\/\/www.nikon.com\/products\/microscope-solutions\/lineup\/integrated\/cl-quant\/."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zahedi, A., On, V., Lin, S.C., Bays, B.C., Omaiye, E., Bhanu, B., and Talbot, P. (2016). Evaluating cell processes, quality, and biomarkers in pluripotent stem cells using video bioinformatics. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0148642"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1109\/TCBB.2014.2306836","article-title":"Bio-driven cell region detection in human embryonic stem cell assay","volume":"11","author":"Guan","year":"2014","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform. (TCBB)"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1016\/j.stemcr.2018.07.016","article-title":"Pluri-IQ: Quantification of Embryonic Stem Cell Pluripotency through an Image-Based Analysis Software","volume":"11","author":"Perestrelo","year":"2018","journal-title":"Stem Cell Rep."},{"key":"ref_17","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Van Valen, D.A., Kudo, T., Lane, K.M., Macklin, D.N., Quach, N.T., DeFelice, M.M., Maayan, I., Tanouchi, Y., Ashley, E.A., and Covert, M.W. (2016). Deep learning automates the quantitative analysis of individual cells in live-cell imaging experiments. PLoS Comput. Biol., 12.","DOI":"10.1371\/journal.pcbi.1005177"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"792","DOI":"10.1016\/j.cell.2018.03.040","article-title":"In silico labeling: Predicting fluorescent labels in unlabeled images","volume":"173","author":"Christiansen","year":"2018","journal-title":"Cell"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1080\/21681163.2016.1149104","article-title":"Microscopy cell counting and detection with fully convolutional regression networks","volume":"6","author":"Xie","year":"2018","journal-title":"Comput. Methods Biomech. Biomed. Eng. Imaging Vis."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"21471","DOI":"10.1038\/srep21471","article-title":"Deep learning in label-free cell classification","volume":"6","author":"Chen","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1240","DOI":"10.1109\/TMI.2016.2538465","article-title":"Brain tumor segmentation using convolutional neural networks in MRI images","volume":"35","author":"Pereira","year":"2016","journal-title":"IEEE Trans. Med. Imaging"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jonnalagedda, P., Schmolze, D., and Bhanu, B. (2018, January 29\u201331). [Regular Paper] MVPNets: Multi-viewing Path Deep Learning Neural Networks for Magnification Invariant Diagnosis in Breast Cancer. Proceedings of the 2018 IEEE 18th International Conference on Bioinformatics and Bioengineering (BIBE), Taichung, Taiwan.","DOI":"10.1109\/BIBE.2018.00044"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"052913","DOI":"10.1117\/1.JBO.26.5.052913","article-title":"Human embryonic stem cell classification: Random network with autoencoded feature extractor","volume":"26","author":"Guan","year":"2021","journal-title":"J. Biomed. Opt."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Witmer, A., and Bhanu, B. (2018, January 20\u201324). Multi-label Classification of Stem Cell Microscopy Images Using Deep Learning. Proceedings of the 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1109\/ICPR.2018.8546040"},{"key":"ref_28","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8\u201313). Generative adversarial nets. Proceedings of the 28th Conference on Neural Information Processing Systems, Montreal, QC, Canada."},{"key":"ref_29","unstructured":"Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv."},{"key":"ref_30","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto."},{"key":"ref_31","unstructured":"Lecun, Y., Cortes, C., and Burges, C. (2021, August 19). The Mnist Database. Available online: http:\/\/yann.lecun.com\/exdb\/mnist\/."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"101552","DOI":"10.1016\/j.media.2019.101552","article-title":"Generative Adversarial Network in Medical Imaging: A Review","volume":"58","author":"Yi","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2720","DOI":"10.1109\/TBME.2018.2814538","article-title":"Medical Image Synthesis with Deep Convolutional Adversarial Networks","volume":"65","author":"Nie","year":"2018","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Majurski, M., Manescu, P., Padi, S., Schaub, N., Hotaling, N., Simon, C., and Bajcsy, P. (2019, January 16\u201320). Cell Image Segmentation Using Generative Adversarial Networks, Transfer Learning, and Augmentations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00145"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"148967","DOI":"10.1109\/ACCESS.2019.2940161","article-title":"Cellular structure image classification with small targeted training samples","volume":"7","author":"Wang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41377-019-0129-y","article-title":"PhaseStain: The digital staining of label-free quantitative phase microscopy images using deep learning","volume":"8","author":"Rivenson","year":"2019","journal-title":"Light. Sci. Appl."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Lee, S., Han, S., Salama, P., Dunn, K.W., and Delp, E.J. (2019, January 8\u201311). Three Dimensional Blind Image Deconvolution for Fluorescence Microscopy using Generative Adversarial Networks. Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, Italy.","DOI":"10.1109\/ISBI.2019.8759250"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bailo, O., Ham, D., and Min Shin, Y. (2019, January 16\u201320). Red Blood Cell Image Generation for Data Augmentation Using Conditional Generative Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00136"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Goldsborough, P., Pawlowski, N., Caicedo, J.C., Singh, S., and Carpenter, A. (2017). CytoGAN: Generative modeling of cell images. bioRxiv, 227645.","DOI":"10.1101\/227645"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Pandhe, N., Rada, B., and Quinn, S. (2018, January 4\u20137). Generative spatiotemporal modeling of neutrophil behavior. Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, DC, USA.","DOI":"10.1109\/ISBI.2018.8363732"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Theagarajan, R., and Bhanu, B. (2019). DeephESC 2.0: Deep Generative Multi Adversarial Networks for improving the classification of hESC. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0212849"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Osokin, A., Chessel, A., Carazo Salas, R.E., and Vaggi, F. (2017, January 22\u201329). Gans for biological image synthesis. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.245"},{"key":"ref_43","first-page":"e13280","article-title":"Improved automatic detection of herpesvirus secondary envelopment stages in electron microscopy by augmenting training data with synthetic labelled images generated by a generative adversarial network","volume":"23","author":"Walther","year":"2021","journal-title":"Cell. Microbiol."},{"key":"ref_44","first-page":"91","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume":"28","author":"Ren","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_45","unstructured":"Shaham, T.R., Dekel, T., and Michaeli, T. (November, January 27). Singan: Learning a generative model from a single natural image. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Korea."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Dimitrakopoulos, P., Sfikas, G., and Nikou, C. (2020, January 3\u20137). ISING-GAN: Annotated data augmentation with a spatially constrained generative adversarial network. Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), Iowa City, IA, USA.","DOI":"10.1109\/ISBI45749.2020.9098618"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Witmer, A., and Bhanu, B. (2018, January 7\u201310). HESCNET: A Synthetically Pre-Trained Convolutional Neural Network for Human Embryonic Stem Cell Colony Classification. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.","DOI":"10.1109\/ICIP.2018.8451624"},{"key":"ref_48","unstructured":"Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN. arXiv."},{"key":"ref_49","unstructured":"Odena, A., Olah, C., and Shlens, J. (2017, January 6\u201311). Conditional image synthesis with auxiliary classifier gans. Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, Sydney, Australia."},{"key":"ref_50","unstructured":"Turner, R., Hung, J., Frank, E., Saatchi, Y., and Yosinski, J. (2019, January 9\u201315). Metropolis-hastings generative adversarial networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_51","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_52","unstructured":"Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved Techniques for Training GANs. arXiv."},{"key":"ref_53","unstructured":"Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2018). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1016\/S0140-6736(07)60111-1","article-title":"Huntington\u2019s disease","volume":"369","author":"Walker","year":"2007","journal-title":"Lancet"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1267","DOI":"10.1056\/NEJM198611133152006","article-title":"Huntingtons disease","volume":"315","author":"Martin","year":"1986","journal-title":"N. Engl. J. Med."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1016\/j.tins.2004.06.008","article-title":"Smoking, nicotine and Parkinson\u2019s disease","volume":"27","author":"Quik","year":"2004","journal-title":"Trends Neurosci."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Karras, T., Laine, S., and Aila, T. (2019, January 16\u201320). A Style-Based Generator Architecture for Generative Adversarial Networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00453"},{"key":"ref_59","unstructured":"Yang, M. (2021, July 19). Imbalanced Dataset Sampler. Available online: https:\/\/github.com\/ufoym\/imbalanced-dataset-sampler."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/206\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:55:10Z","timestamp":1760169310000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/1\/206"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,29]]},"references-count":59,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22010206"],"URL":"https:\/\/doi.org\/10.3390\/s22010206","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,12,29]]}}}