{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T13:07:32Z","timestamp":1789477652828,"version":"build-2803163510"},"reference-count":29,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T00:00:00Z","timestamp":1789430400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Multi-modal conditioning in latent diffusion models\u2014combining text, structural, and spatial guidance signals\u2014substantially improves controllable image synthesis, yet two limitations persist across most existing frameworks. First, conditioning modalities are fused using fixed architectural weights that remain constant regardless of the input, preventing any input-adaptive rebalancing of guidance. Second, most existing multi-modal approaches are designed under the assumption that all conditioning signals will be present at inference time, leaving the model without any way to degrade gracefully when some modalities are unavailable. This paper introduces a unified latent diffusion framework that addresses both limitations simultaneously.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      Our architecture integrates three complementary conditioning modalities\u2014semantic text embeddings via CLIP, structural boundary maps derived from Canny edge detection, and regional semantic layout from segmentation masks-as co\u2014equal inputs into a single U-Net denoising backbone. The framework has two core components: Softmax-Normalized Adaptive Guidance Fusion (SNAGF), which replaces fixed fusion weights with three learnable, softmax-normalized modality importance scores optimised end-to-end alongside the diffusion objective; and Modality-Dropout Training (MDT), a structured regularisation strategy that randomly zeroes individual guidance representations with probability\n                      <jats:italic>p<\/jats:italic>\n                      <jats:sub>drop<\/jats:sub>\n                      = 0.3 during training, training the model to produce coherent outputs under any available subset of conditioning inputs.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Evaluated on CIFAR-10, the full SNAGF+MDT framework achieves SSIM 0.93, FID 17.2, PSNR 34.1 dB, CLIP Score 0.88, LPIPS 0.09, and IS 10.4. MDT alone yields an average 15.5% FID improvement across partial-conditioning inference configurations, and learned SNAGF weights converge to \u03bb\n                      <jats:sub>text<\/jats:sub>\n                      = 0.370, \u03bb\n                      <jats:sub>edge<\/jats:sub>\n                      = 0.363, \u03bb\n                      <jats:sub>seg<\/jats:sub>\n                      = 0.267, demonstrating stable convergence from epoch 30 onward. Results are averaged over three random seeds.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Taken together, they indicate that adaptive modality weighting outperforms fixed conditioning strategies and that modality-dropout training provides an effective partial-conditioning robustness mechanism. All findings are established at 32 \u00d7 32 resolution on a single benchmark with algorithmically constructed conditioning signals; validation on higher-resolution datasets with naturally paired text and structural annotation remains necessary before these conclusions can be generalised.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2026.1867917","type":"journal-article","created":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T12:31:39Z","timestamp":1789475499000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["A unified multi-modal latent diffusion framework with modality-dropout training"],"prefix":"10.3389","volume":"9","author":[{"given":"S.","family":"Remya","sequence":"first","affiliation":[{"name":"Department of CSE, Amrita School of Computing, Amrita Vishwa Vidyapeetham, Amritapuri","place":["Kollam, Kerala, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Manu J.","family":"Pillai","sequence":"additional","affiliation":[{"name":"Department of CSE, TKM College of Engineering","place":["Kollam, Kerala, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laveena","family":"Herman","sequence":"additional","affiliation":[{"name":"Department of CSE, TKM College of Engineering","place":["Kollam, Kerala, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"B.","family":"Niranjan","sequence":"additional","affiliation":[{"name":"Department of CSE, TKM College of Engineering","place":["Kollam, Kerala, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Somula","family":"Ramasubbareddy","sequence":"additional","affiliation":[{"name":"Symbiosis Institute of Technology, Hyderabad Campus, Symbiosis International (Deemed University)","place":["Pune, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong Yun","family":"Cho","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Engineering, Sunchon National University","place":["Suncheon, Jeollanam-do, Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,9,15]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: a survey and taxonomy","volume":"41","author":"Baltru\u0161aitis","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B2","article-title":"Stable video diffusion: Scaling latent video diffusion models to large datasets","author":"Blattmann","year":"2023","journal-title":"arXiv"},{"key":"B3","article-title":"Generated faces in the wild: quantitative comparison of stable diffusion, midjourney and DALL-E 2","author":"Borji","year":"2022","journal-title":"arXiv"},{"key":"B4","author":"Brock","year":"2019"},{"key":"B5","first-page":"1692","article-title":"\u201cMasked image training for generalizable deep image denoising,\u201d","volume-title":"Proceedings of IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Chen","year":"2023"},{"key":"B6","first-page":"2672","article-title":"\u201cGenerative adversarial nets,\u201d","author":"Goodfellow","year":"2014"},{"key":"B7","first-page":"6626","article-title":"\u201cGANs trained by a two time-scale update rule converge to a local Nash equilibrium,\u201d","author":"Heusel","year":"2017","journal-title":"Proceedings of Advances in Neural Information Processing Systems"},{"key":"B8","first-page":"6840","author":"Ho","year":"2020"},{"key":"B9","article-title":"\u201cClassifier-free diffusion guidance,\u201d","author":"Ho","year":"2022","journal-title":"Proceedings of NeurIPS Workshop on Deep Generative Models"},{"key":"B10","first-page":"1125","article-title":"\u201cImage-to-image translation with conditional adversarial networks,\u201d","volume-title":"Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Isola","year":"2017"},{"key":"B11","first-page":"4401","article-title":"\u201cA style-based generator architecture for generative adversarial networks,\u201d","volume-title":"Proceedings of Conference on Computer Vision and Pattern Recognition","author":"Karras","year":"2019"},{"key":"B12","first-page":"14","article-title":"\u201cAuto-Encoding Variational Bayes,\u201d","volume-title":"Proceedings of International Conference on Learning Representations","author":"Kingma","year":"2014"},{"key":"B13","article-title":"On fast sampling of diffusion probabilistic models","author":"Kong","year":"2021","journal-title":"arXiv"},{"key":"B14","author":"Krizhevsky","year":"2009","journal-title":"Learning multiple layers of features from tiny"},{"key":"B15","doi-asserted-by":"publisher","first-page":"2302","DOI":"10.1609\/aaai.v35i3.16330","article-title":"SMIL: multimodal learning with severely missing modality","volume":"35","author":"Ma","year":"2021","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B16","doi-asserted-by":"publisher","first-page":"1692","DOI":"10.1109\/TPAMI.2015.2461544","article-title":"ModDrop: adaptive multi-modal gesture recognition","volume":"38","author":"Neverova","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B17","first-page":"16784","article-title":"\u201cGLIDE: towards photorealistic image generation with text-guided diffusion models,\u201d","author":"Nichol","year":"2022","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"B18","first-page":"8748","article-title":"\u201cLearning transferable visual models from natural language supervision,\u201d","author":"Radford","year":"2021","journal-title":"Proceedings of International Conference on Machine Learning"},{"key":"B19","author":"Radford","year":"2016"},{"key":"B20","first-page":"8821","article-title":"\u201cZero-shot text-to-image generation,\u201d","author":"Ramesh","year":"2021","journal-title":"Proceedings of the 38th International Conference on Machine Learning"},{"key":"B21","article-title":"\u201cGenerating diverse high-fidelity images with VQ-VAE-2,\u201d","volume-title":"Proceedings of Advances in Neural Information Processing Systems","author":"Razavi","year":"2019"},{"key":"B22","doi-asserted-by":"publisher","first-page":"10684","DOI":"10.1109\/CVPR52688.2022.01042","author":"Rombach","year":"2022"},{"key":"B23","doi-asserted-by":"publisher","first-page":"36479","DOI":"10.52202\/068431-2643","article-title":"\u201cPhotorealistic text-to-image diffusion models with deep language understanding,\u201d","author":"Saharia","year":"2022","journal-title":"Proceeding of Advances in Neural Information Processing Systems"},{"key":"B24","author":"Song","year":"2021"},{"key":"B25","first-page":"1929","article-title":"\u201cDropout: a simple way to prevent neural networks from overfitting,\u201d","volume":"15","author":"Srivastava","year":"2014"},{"key":"B26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3592451","article-title":"UniTune: text-driven image editing by fine tuning a diffusion model","volume":"42","author":"Valevski","year":"2023","journal-title":"ACM Trans.Graph"},{"key":"B27","first-page":"507","article-title":"\u201cP-2.9: a review of image generation methods based on deep learning,\u201d","volume-title":"Proceedings of SID Symposium Digest of Technical Papers","author":"Wang","year":"2023"},{"key":"B28","first-page":"7732","article-title":"\u201cHarnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis,\u201d","volume-title":"Proceedings of IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Wu","year":"2023"},{"key":"B29","doi-asserted-by":"publisher","first-page":"3836","DOI":"10.1109\/ICCV51070.2023.00355","article-title":"\u201cAdding conditional control to text-to-image diffusion models,\u201d","author":"Zhang","year":"2023"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1867917\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T12:31:42Z","timestamp":1789475502000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2026.1867917\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,15]]},"references-count":29,"alternative-id":["10.3389\/frai.2026.1867917"],"URL":"https:\/\/doi.org\/10.3389\/frai.2026.1867917","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,15]]},"article-number":"1867917"}}