{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,25]],"date-time":"2026-10-25T03:36:41Z","timestamp":1792899401603,"version":"4.5.0"},"reference-count":58,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T00:00:00Z","timestamp":1631145600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurosci."],"abstract":"<jats:p>The infant brain undergoes a remarkable period of neural development that is crucial for the development of cognitive and behavioral capacities (Hasegawa et al., <jats:xref>2018<\/jats:xref>). Longitudinal magnetic resonance imaging (MRI) is able to characterize the developmental trajectories and is critical in neuroimaging studies of early brain development. However, missing data at different time points is an unavoidable occurrence in longitudinal studies owing to participant attrition and scan failure. Compared to dropping incomplete data, data imputation is considered a better solution to address such missing data in order to preserve all available samples. In this paper, we adapt generative adversarial networks (GAN) to a new application: longitudinal image prediction of structural MRI in the first year of life. In contrast to existing medical image-to-image translation applications of GANs, where inputs and outputs share a very close anatomical structure, our task is more challenging as brain size, shape and tissue contrast vary significantly between the input data and the predicted data. Several improvements over existing GAN approaches are proposed to address these challenges in our task. To enhance the realism, crispness, and accuracy of the predicted images, we incorporate both a traditional voxel-wise reconstruction loss as well as a perceptual loss term into the adversarial learning scheme. As the differing contrast changes in T1w and T2w MR images in the first year of life, we incorporate multi-contrast images leading to our proposed 3D multi-contrast perceptual adversarial network (MPGAN). Extensive evaluations are performed to assess the qualityand fidelity of the predicted images, including qualitative and quantitative assessments of the image appearance, as well as quantitative assessment on two segmentation tasks. Our experimental results show that our MPGAN is an effective solution for longitudinal MR image data imputation in the infant brain. We further apply our predicted\/imputed images to two practical tasks, a regression task and a classification task, in order to highlight the enhanced task-related performance following image imputation. The results show that the model performance in both tasks is improved by including the additional imputed data, demonstrating the usability of the predicted images generated from our approach.<\/jats:p>","DOI":"10.3389\/fnins.2021.653213","type":"journal-article","created":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T11:36:59Z","timestamp":1631187419000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Longitudinal Prediction of Infant MR Images With Multi-Contrast Perceptual Adversarial Learning"],"prefix":"10.3389","volume":"15","author":[{"given":"Liying","family":"Peng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lanfen","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yusen","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yen-wei","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhanhao","family":"Mo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roza M.","family":"Vlasova","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sun Hyung","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alan C.","family":"Evans","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen R.","family":"Dager","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annette M.","family":"Estes","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert C.","family":"McKinstry","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kelly N.","family":"Botteron","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guido","family":"Gerig","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robert T.","family":"Schultz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heather C.","family":"Hazlett","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph","family":"Piven","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Catherine A.","family":"Burrows","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebecca L.","family":"Grzadzinski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jessica B.","family":"Girault","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark D.","family":"Shen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin A.","family":"Styner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2021,9,9]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"101684","DOI":"10.1016\/j.compmedimag.2019.101684","article-title":"MedGAN: medical image translation using GANs","volume":"79","author":"Armanious","year":"2020","journal-title":"Comput. Med. Imaging Graph"},{"key":"B2","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1007\/978-3-319-68127-6_6","article-title":"Virtual pet images from ct data using deep convolutional networks: initial results","volume-title":"International Workshop on Simulation and Synthesis in Medical Imaging","author":"Ben-Cohen","year":"2017"},{"key":"B3","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/978-3-319-67564-0_5","article-title":"Synthesis of positron emission tomography (PET) images via multi-channel generative adversarial networks (GANs)","volume-title":"Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment","author":"Bi","year":"2017"},{"key":"B4","doi-asserted-by":"crossref","DOI":"10.1117\/12.2293256","article-title":"Modelling the progression of Alzheimer's disease in MRI using generative adversarial networks","volume-title":"Medical Imaging 2018: Image Processing","author":"Bowles","year":"2018"},{"key":"B5","doi-asserted-by":"crossref","first-page":"785","DOI":"10.1145\/2939672.2939785","article-title":"Xgboost: a scalable tree boosting system","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Chen","year":"2016"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1111","DOI":"10.2967\/jnumed.117.199414","article-title":"Generation of structural MR images from amyloid pet: application to MR-less quantification","volume":"59","author":"Choi","year":"2018","journal-title":"J. Nuclear Med"},{"key":"B7","first-page":"424","article-title":"3D U-Net: learning dense volumetric segmentation from sparse annotation","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"\u00c7i\u00e7ek","year":"2016"},{"key":"B8","doi-asserted-by":"publisher","first-page":"2375","DOI":"10.1109\/TMI.2019.2901750","article-title":"Image synthesis in multi-contrast MRI with conditional generative adversarial networks","volume":"38","author":"Dar","year":"2019","journal-title":"IEEE Trans. Med. Imaging"},{"key":"B9","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1109\/TMM.2020.2975961","article-title":"SPA-GAN: spatial attention GAN for image-to-image translation","volume":"23","author":"Emami","year":"2021","journal-title":"IEEE Transactions on Multimedia"},{"key":"B10","doi-asserted-by":"crossref","first-page":"718","DOI":"10.1007\/978-3-642-38868-2_60","article-title":"Geodesic shape regression in the framework of currents","volume-title":"International Conference on Information Processing in Medical Imaging","author":"Fishbaugh","year":"2013"},{"key":"B11","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1109\/ISBI.2014.6867889","article-title":"Geodesic regression of image and shape data for improved modeling of 4D trajectories","volume-title":"2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI)","author":"Fishbaugh","year":"2014"},{"key":"B12","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1007\/s11263-012-0591-y","article-title":"Geodesic regression and the theory of least squares on riemannian manifolds","volume":"105","author":"Fletcher","year":"2013","journal-title":"Int. J. Comput. Vision"},{"key":"B13","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1038\/nrn.2018.1","article-title":"Imaging structural and functional brain development in early childhood","volume":"19","author":"Gilmore","year":"2018","journal-title":"Nat. Rev. Neurosci"},{"key":"B14","doi-asserted-by":"publisher","first-page":"2478","DOI":"10.1093\/cercor\/bhr327","article-title":"Longitudinal development of cortical and subcortical gray matter from birth to 2 years","volume":"22","author":"Gilmore","year":"2012","journal-title":"Cereb. Cortex"},{"key":"B15","first-page":"2672","article-title":"Generative adversarial nets","volume-title":"Advances in Neural Information Processing Systems 27","author":"Goodfellow","year":"2014"},{"key":"B16","doi-asserted-by":"publisher","first-page":"566","DOI":"10.3389\/fnins.2018.00566","article-title":"Developmental trajectory of infant brain signal variability: a longitudinal pilot study","volume":"12","author":"Hasegawa","year":"2018","journal-title":"Front. Neurosci"},{"key":"B17","doi-asserted-by":"publisher","first-page":"348","DOI":"10.1038\/nature21369","article-title":"Early brain development in infants at high risk for autism spectrum disorder","volume":"542","author":"Hazlett","year":"2017","journal-title":"Nature"},{"key":"B18","first-page":"172","article-title":"Multimodal unsupervised image-to-image translation","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"Huang","year":"2018"},{"key":"B19","doi-asserted-by":"publisher","first-page":"2400","DOI":"10.1007\/s10803-012-1719-1","article-title":"Standardizing ados domain scores: separating severity of social affect and restricted and repetitive behaviors","volume":"44","author":"Hus","year":"2014","journal-title":"J. Autism Dev. Disord"},{"key":"B20","first-page":"1125","article-title":"Image-to-image translation with conditional adversarial networks","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Isola","year":"2017"},{"key":"B21","doi-asserted-by":"publisher","first-page":"2361","DOI":"10.3390\/s19102361","article-title":"Deep CT to MR synthesis using paired and unpaired data","volume":"19","author":"Jin","year":"2019","journal-title":"Sensors"},{"key":"B22","first-page":"694","article-title":"Perceptual losses for real-time style transfer and super-resolution","volume-title":"European Conference on Computer Vision","author":"Johnson","year":"2016"},{"key":"B23","article-title":"Progressive growing of gans for improved quality, stability, and variation","author":"Karras","year":"2017","journal-title":"arXiv [preprint]. arXiv:1710.10196"},{"key":"B24","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2014","journal-title":"arXiv [preprint]. arXiv:1412.6980"},{"key":"B25","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1002\/sim.4780070131","article-title":"Missing data in longitudinal studies","volume":"7","author":"Laird","year":"1988","journal-title":"Stat. Med"},{"key":"B26","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1007\/978-3-319-70139-4","article-title":"Unsupervised image-to-image translation networks","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems","author":"Liu","year":"2017"},{"key":"B27","volume-title":"Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) Manual (Part I): Modules 1\u20134","author":"Lord","year":"2012"},{"key":"B28","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1016\/j.dcn.2017.10.001","article-title":"Making an unknown unknown a known unknown: missing data in longitudinal neuroimaging studies","volume":"33","author":"Matta","year":"2018","journal-title":"Dev. Cogn. Neurosci"},{"key":"B29","doi-asserted-by":"publisher","first-page":"2865","DOI":"10.1002\/hbm.23555","article-title":"Can we predict subject-specific dynamic cortical thickness maps during infancy from birth?","volume":"38","author":"Meng","year":"2017","journal-title":"Hum. Brain Mapp"},{"key":"B30","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1073\/pnas.36.1.48","article-title":"Equilibrium points in n-person games","volume":"36","author":"Nash","year":"1950","journal-title":"Proc. Natl. Acad. Sci. U.S.A"},{"key":"B31","first-page":"417","article-title":"Medical image synthesis with context-aware generative adversarial networks","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Nie","year":"2017"},{"key":"B32","first-page":"655","article-title":"Geodesic regression for image time-series","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Niethammer","year":"2011"},{"key":"B33","first-page":"137","article-title":"Disease-image specific generative adversarial network for brain disease diagnosis with incomplete multi-modal neuroimages","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Pan","year":"2019"},{"key":"B34","first-page":"455","article-title":"Synthesizing missing pet from mri with cycle-consistent generative adversarial networks for Alzheimer's disease diagnosis","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Pan","year":"2018"},{"key":"B35","first-page":"2536","article-title":"Context encoders: feature learning by inpainting","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Pathak","year":"2016"},{"key":"B36","first-page":"786","article-title":"Wavelet-based semi-supervised adversarial learning for synthesizing realistic 7T from 3T MRI","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Qu","year":"2019"},{"key":"B37","volume-title":"Python Machine Learning","author":"Raschka","year":"2015"},{"key":"B38","first-page":"164","article-title":"Degenerative adversarial neuroimage nets: generating images that mimic disease progression","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Ravi","year":"2019"},{"key":"B39","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.media.2015.10.007","article-title":"Predicting infant cortical surface development using a 4D varifold-based learning framework and local topography-based shape morphing","volume":"28","author":"Rekik","year":"2016","journal-title":"Med. Image Anal"},{"key":"B40","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1007\/978-3-319-28194-0_24","article-title":"Prediction of infant mri appearance and anatomical structure evolution using sparse patch-based metamorphosis learning framework","volume-title":"International Workshop on Patch-based Techniques in Medical Imaging","author":"Rekik","year":"2015"},{"key":"B41","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1097\/WCO.0000000000000713","article-title":"The atypical social brain network in autism: advances in structural and functional MRI studies","volume":"32","author":"Sato","year":"2019","journal-title":"Curr. Opin. Neurol"},{"key":"B42","first-page":"560","article-title":"A hierarchical geodesic model for diffeomorphic longitudinal shape analysis","volume-title":"International Conference on Information Processing in Medical Imaging","author":"Singh","year":""},{"key":"B43","first-page":"1219","article-title":"A vector momenta formulation of diffeomorphisms for improved geodesic regression and atlas construction","volume-title":"2013 IEEE 10th International Symposium on Biomedical Imaging","author":"Singh","year":""},{"key":"B44","doi-asserted-by":"publisher","first-page":"3979","DOI":"10.1007\/s00429-015-1140-6","article-title":"Identification of progressive mild cognitive impairment patients using incomplete longitudinal MRI scans","volume":"221","author":"Thung","year":"2016","journal-title":"Brain Struct. Funct"},{"key":"B45","article-title":"Instance normalization: the missing ingredient for fast stylization","author":"Ulyanov","year":"2016","journal-title":"arXiv [preprint]. arXiv:1607.08022"},{"key":"B46","first-page":"7","article-title":"3D segmentation in the clinic: a grand challenge","volume-title":"MICCAI Workshop on 3D Segmentation in the Clinic: A Grand Challenge","author":"Van Ginneken","year":"2007"},{"key":"B47","doi-asserted-by":"publisher","first-page":"7","DOI":"10.3389\/fninf.2014.00007","article-title":"Multi-atlas segmentation of subcortical brain structures via the autoseg software pipeline","volume":"8","author":"Wang","year":"2014","journal-title":"Front. Neuroinform"},{"key":"B48","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1007\/978-3-319-68127-6_2","article-title":"Deep MR to CT synthesis using unpaired data","volume-title":"International Workshop on Simulation and Synthesis in Medical Imaging","author":"Wolterink","year":"2017"},{"key":"B49","doi-asserted-by":"publisher","first-page":"102169","DOI":"10.1016\/j.media.2021.102169","article-title":"Learning to synthesise the ageing brain without longitudinal data","volume":"73","author":"Xia","year":"2019","journal-title":"Med. Image Anal"},{"key":"B50","doi-asserted-by":"publisher","first-page":"126651","DOI":"10.1109\/ACCESS.2019.2939654","article-title":"Consistent embedded GAN for image-to-image translation","volume":"7","author":"Xiong","year":"2019","journal-title":"IEEE Access"},{"key":"B51","first-page":"1","article-title":"MRI cross-modality image-to-image translation","volume":"10","author":"Yang","year":"2020","journal-title":"Sci. Rep"},{"key":"B52","first-page":"2849","article-title":"DualGAN: unsupervised dual learning for image-to-image translation","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Yi","year":"2017"},{"key":"B53","first-page":"7354","article-title":"Self-attention generative adversarial networks","volume-title":"International Conference on Machine Learning","author":"Zhang","year":"2019"},{"key":"B54","first-page":"586","article-title":"The unreasonable effectiveness of deep features as a perceptual metric","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Zhang","year":"2018"},{"key":"B55","first-page":"475","article-title":"Harmonization of infant cortical thickness using surface-to-surface cycle-consistent adversarial networks","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Zhao","year":"2019"},{"key":"B56","article-title":"Loss functions for neural networks for image processing","author":"Zhao","year":"2015","journal-title":"arXiv [preprint]. arXiv:1511.08861"},{"key":"B57","first-page":"384","article-title":"Models genesis: generic autodidactic models for 3D medical image analysis","volume-title":"International Conference on Medical Image Computing and Computer-Assisted Intervention","author":"Zhou","year":"2019"},{"key":"B58","first-page":"2223","article-title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Zhu","year":"2017"}],"container-title":["Frontiers in Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2021.653213\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T11:37:53Z","timestamp":1631187473000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2021.653213\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,9]]},"references-count":58,"alternative-id":["10.3389\/fnins.2021.653213"],"URL":"https:\/\/doi.org\/10.3389\/fnins.2021.653213","relation":{},"ISSN":["1662-453X"],"issn-type":[{"value":"1662-453X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,9]]},"article-number":"653213"}}