{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T16:36:38Z","timestamp":1769013398900,"version":"3.49.0"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"30","license":[{"start":{"date-parts":[[2025,2,5]],"date-time":"2025-02-05T00:00:00Z","timestamp":1738713600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,5]],"date-time":"2025-02-05T00:00:00Z","timestamp":1738713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100020043","name":"Yeditepe University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100020043","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Multimed Tools Appl"],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>The proliferation of extensive neural network architectures, particularly deep learning models, presents a challenge in terms of resource-intensive training. GPU memory constraints have become a notable bottleneck in training such sizable models. Existing strategies, including data parallelism, model parallelism, pipeline parallelism, and fully sharded data parallelism, offer partial solutions. Model parallelism, in particular, enables the distribution of the entire model across multiple GPUs, yet the ensuing data communication between these partitions slows down training. Instead of using the entire model for training, this study advocates partitioning the model across GPUs and generating synthetic intermediate labels to train individual segments. These labels, produced through a random process, mitigate memory overhead and computational load. This approach results in a more efficient training process that minimizes data communication while maintaining model accuracy. The method is validated using 6-layer fully-connected networks, via the extended MNIST, CIFAR10 and CIFAR100 datasets. It is shown that the computational improvement to reach 90% of the cross-yield accuracy can be as high as 66%. Additionally, the improvement in training bandwidth compared to standard model parallelism is quantitatively demonstrated through an example scenario. This work contributes to mitigating the resource-intensive nature of training large neural networks, paving the way for more efficient deep learning model development.<\/jats:p>","DOI":"10.1007\/s11042-025-20666-9","type":"journal-article","created":{"date-parts":[[2025,2,5]],"date-time":"2025-02-05T06:28:40Z","timestamp":1738736920000},"page":"36899-36918","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Partitioned neural network training via synthetic intermediate labels"],"prefix":"10.1007","volume":"84","author":[{"given":"Cevat Volkan","family":"Karadag","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0525-8900","authenticated-orcid":false,"given":"Nezih","family":"Topaloglu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,2,5]]},"reference":[{"key":"20666_CR1","doi-asserted-by":"publisher","unstructured":"Sutton DP, Carlisle MC, Sarmiento TA, Baird LC (2009) Partitioned neural networks. In: 2009 International joint conference on neural networks. pp 3032\u20133037. https:\/\/doi.org\/10.1109\/IJCNN.2009.5178994","DOI":"10.1109\/IJCNN.2009.5178994"},{"key":"20666_CR2","doi-asserted-by":"publisher","unstructured":"Shriram S.B., Garg, A., Kulkarni, P (2019) Dynamic memory management for gpu-based training of deep neural networks. In: 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS). pp 200\u2013209 https:\/\/doi.org\/10.1109\/IPDPS.2019.00030","DOI":"10.1109\/IPDPS.2019.00030"},{"issue":"11","key":"20666_CR3","doi-asserted-by":"publisher","first-page":"12741","DOI":"10.1007\/s11227-021-03746-z","volume":"77","author":"H Fu","year":"2021","unstructured":"Fu H, Tang S, He B, Yu C, Sun J (2021) Hgp4cnn: an efficient parallelization framework for training convolutional neural networks on modern gpus. J Supercomput 77(11):12741\u201312770. https:\/\/doi.org\/10.1007\/s11227-021-03746-z","journal-title":"J Supercomput"},{"issue":"5","key":"20666_CR4","doi-asserted-by":"publisher","first-page":"1012","DOI":"10.1007\/s12083-017-0574-4","volume":"11","author":"Q Yao","year":"2018","unstructured":"Yao Q, Liao X, Jin H (2018) Training deep neural network on multiple gpus with a model averaging method. Peer-to-Peer Netw Appl 11(5):1012\u20131021. https:\/\/doi.org\/10.1007\/s12083-017-0574-4","journal-title":"Peer-to-Peer Netw Appl"},{"key":"20666_CR5","unstructured":"Meng C, Sun M, Yang J, Qiu M, Gu Y (2017) Training deeper models by gpu memory optimization on tensorflow. https:\/\/api.semanticscholar.org\/CorpusID:23312642"},{"key":"20666_CR6","unstructured":"BFloat16: The secret to high performance on Cloud TPUs. https:\/\/cloud.google.com\/blog\/products\/ai-machine-learning\/bfloat16-the-secret-to-high-performance-on-cloud-tpus. Accessed on 22 July 2023"},{"key":"20666_CR7","unstructured":"Jia Z, Zaharia M, Aiken A (2018) Beyond data and model parallelism for deep neural networks"},{"key":"20666_CR8","unstructured":"Getting Started with Distributed Data Parallel. https:\/\/github.com\/pytorch\/tutorials\/blob\/main\/intermediate_source\/ddp_tutorial.rst. Accessed on 22 July 2023"},{"issue":"12","key":"20666_CR9","doi-asserted-by":"publisher","first-page":"3005","DOI":"10.14778\/3415478.3415530","volume":"13","author":"S Li","year":"2020","unstructured":"Li S, Zhao Y, Varma R, Salpekar O, Noordhuis P, Li T, Paszke A, Smith J, Vaughan B, Damania P, Chintala S (2020) Pytorch distributed: experiences on accelerating data parallel training. Proc VLDB Endow 13(12):3005\u20133018. https:\/\/doi.org\/10.14778\/3415478.3415530","journal-title":"Proc VLDB Endow"},{"key":"20666_CR10","volume-title":"GPipe: efficient training of giant neural networks using pipeline parallelism","author":"Y Huang","year":"2019","unstructured":"Huang Y, Cheng Y, Bapna A, Firat O, Chen MX, Chen D, Lee H, Ngiam J, Le QV, Wu Y, Chen Z (2019) GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA"},{"key":"20666_CR11","doi-asserted-by":"publisher","unstructured":"Castell\u00f3 A, Dolz MF, Quintana-Ort\u00ed ES, Duato J (2019) Analysis of model parallelism for distributed neural networks. In: Proceedings of the 26th European MPI Users\u2019 Group Meeting. EuroMPI \u201919. Association for Computing Machinery, New York, NY, USA. https:\/\/doi.org\/10.1145\/3343211.3343218","DOI":"10.1145\/3343211.3343218"},{"issue":"8","key":"20666_CR12","doi-asserted-by":"publisher","first-page":"11587","DOI":"10.1007\/s11042-022-12292-6","volume":"81","author":"L Wanwu","year":"2022","unstructured":"Wanwu L, Lin L, Jixian Z, Shuai L, Jiahao Q (2022) Multi-core parallel architecture design and experiment for deep learning model training. Multimed Tools Appl 81(8):11587\u201311604. https:\/\/doi.org\/10.1007\/s11042-022-12292-6","journal-title":"Multimed Tools Appl"},{"key":"20666_CR13","doi-asserted-by":"crossref","unstructured":"Rajbhandari S, Rasley J, Ruwase O, He Y (2020) ZeRO: memory optimizations toward training trillion parameter models","DOI":"10.1109\/SC41405.2020.00024"},{"key":"20666_CR14","doi-asserted-by":"crossref","unstructured":"Zhao Y, Gu A, Varma R, Luo L, Huang C-C, Xu M, Wright L, Shojanazeri H, Ott M, Shleifer S, Desmaison A, Balioglu C, Nguyen B, Chauhan G, Hao Y, Li S (2023) PyTorch FSDP: experiences on scaling fully sharded data parallel","DOI":"10.14778\/3611540.3611569"},{"key":"20666_CR15","unstructured":"Mlodozeniec B, Reisser M, Louizos C (2023) Hyperparameter optimization through neural network partitioning"},{"key":"20666_CR16","doi-asserted-by":"crossref","unstructured":"Akintoye SB, Han L, Lloyd H, Zhang X, Dancey D, Chen H, Zhang D (2022) Layer-wise partitioning and merging for efficient and scalable deep learning","DOI":"10.2139\/ssrn.4279818"},{"key":"20666_CR17","unstructured":"Liao R, Brockschmidt M, Tarlow D, Gaunt AL, Urtasun R, Zemel R (2018) Graph partition neural networks for semi-supervised classification"},{"key":"20666_CR18","doi-asserted-by":"publisher","unstructured":"Na J, Zhang H, Lian J, Zhang B (2022) Partitioning dnns for optimizing distributed inference performance on cooperative edge devices: a genetic algorithm approach. Appl Sci 12(20). https:\/\/doi.org\/10.3390\/app122010619","DOI":"10.3390\/app122010619"},{"key":"20666_CR19","doi-asserted-by":"publisher","unstructured":"Oliveira F, Borin E (2019) Partitioning convolutional neural networks to maximize the inference rate on constrained iot devices. Fut Int 11(10). https:\/\/doi.org\/10.3390\/fi11100209","DOI":"10.3390\/fi11100209"},{"key":"20666_CR20","unstructured":"Parthasarathy A, Krishnamachari B (2023) Partitioning and deployment of deep neural networks on edge clusters"},{"key":"20666_CR21","doi-asserted-by":"publisher","unstructured":"Jain A, Phanishayee A, Mars J, Tang L, Pekhimenko G (2018) Gist: Efficient data encoding for deep neural network training. In: 2018 ACM\/IEEE 45th annual International Symposium on Computer Architecture (ISCA). pp 776\u2013789. https:\/\/doi.org\/10.1109\/ISCA.2018.00070","DOI":"10.1109\/ISCA.2018.00070"},{"key":"20666_CR22","doi-asserted-by":"publisher","unstructured":"Wang L, Ye J, Zhao Y, Wu W, Li A, Song SL, Xu Z, Kraska T (2018) Superneurons. In: Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. ACM. https:\/\/doi.org\/10.1145\/3178487.3178491","DOI":"10.1145\/3178487.3178491"},{"key":"20666_CR23","unstructured":"Anil R, Gupta V, Koren T, Singer Y (2019) Memory-efficient adaptive optimization"},{"key":"20666_CR24","unstructured":"Shazeer N, Stern M (2018) Adafactor: adaptive learning rates with sublinear memory cost"},{"issue":"2","key":"20666_CR25","doi-asserted-by":"publisher","first-page":"6249","DOI":"10.1007\/s11042-023-15747-6","volume":"83","author":"IH Rather","year":"2024","unstructured":"Rather IH, Kumar S (2024) Generative adversarial network based synthetic data training model for lightweight convolutional neural networks. Multimed Tools Appl 83(2):6249\u20136271. https:\/\/doi.org\/10.1007\/s11042-023-15747-6","journal-title":"Multimed Tools Appl"},{"issue":"5","key":"20666_CR26","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1007\/s11633-022-1411-7","volume":"21","author":"A Mumuni","year":"2024","unstructured":"Mumuni A, Mumuni F, Gerrar NK (2024) A survey of synthetic data augmentation methods in machine vision. Mach Intell Res 21(5):831\u2013869. https:\/\/doi.org\/10.1007\/s11633-022-1411-7","journal-title":"Mach Intell Res"},{"key":"20666_CR27","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1007\/978-3-319-59063-9_9","volume-title":"Artificial intelligence and soft computing","author":"J Kolbusz","year":"2017","unstructured":"Kolbusz J, Rozycki P, Wilamowski BM (2017) The study of architecture mlp with linear neurons in order to eliminate the \u201cvanishing gradient\u2019\u2019 problem. In: Rutkowski L, Korytkowski M, Scherer R, Tadeusiewicz R, Zadeh LA, Zurada JM (eds) Artificial intelligence and soft computing. Springer, Cham, pp 97\u2013106"},{"key":"20666_CR28","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1007\/978-3-030-57675-2_10","volume-title":"Euro-Par 2020: parallel processing","author":"O Beaumont","year":"2020","unstructured":"Beaumont O, Eyraud-Dubois L, Shilova A (2020) Optimal gpu-cpu offloading strategies for deep neural network training. In: Malawski M, Rzadca K (eds) Euro-Par 2020: parallel processing. Springer, Cham, pp 151\u2013166"},{"key":"20666_CR29","unstructured":"Maennel H, Alabdulmohsin I, Tolstikhin I, Baldock RJN, Bousquet O, Gelly S, Keysers D (2020) What do neural networks learn when trained with random labels? In: Proceedings of the 34th international conference on Neural Information Processing Systems. NIPS\u201920. Curran Associates Inc., Red Hook, NY, USA"},{"key":"20666_CR30","doi-asserted-by":"publisher","unstructured":"Franchi G, Bursuc A, Aldea E, Dubuisson S, Bloch I (2020) Tradi: Tracking deep neural network weight distributions. In: Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII. Springer, Berlin, Heidelberg, pp 105\u2013121. https:\/\/doi.org\/10.1007\/978-3-030-58520-4_7","DOI":"10.1007\/978-3-030-58520-4_7"},{"key":"20666_CR31","doi-asserted-by":"crossref","unstructured":"Cohen G, Afshar S, Tapson J, Schaik A (2017) EMNIST: an extension of MNIST to handwritten letters","DOI":"10.1109\/IJCNN.2017.7966217"},{"issue":"6","key":"20666_CR32","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/MSP.2012.2211477","volume":"29","author":"L Deng","year":"2012","unstructured":"Deng L (2012) The mnist database of handwritten digit images for machine learning research. IEEE Signal Process Mag 29(6):141\u2013142","journal-title":"IEEE Signal Process Mag"},{"key":"20666_CR33","unstructured":"Linear \u2013 PyTorch 2.0 Documentation. https:\/\/pytorch.org\/docs\/stable\/generated\/torch.nn.Linear.html#torch.nn.Linear. Accessed on 15 Aug 2023"},{"key":"20666_CR34","unstructured":"Zhang C, Bengio S, Hardt M, Recht B, Vinyals O (2017) Understanding deep learning requires rethinking generalization"},{"key":"20666_CR35","doi-asserted-by":"publisher","unstructured":"Jain A, Awan AA, Aljuhani AM, Hashmi JM, Anthony QG, Subramoni H, Panda DK, Machiraju R, Parwani A (2020) Gems: Gpu-enabled memory-aware model-parallelism system for distributed dnn training. In: SC20: international conference for high performance computing, networking, storage and analysis. pp 1\u201315. https:\/\/doi.org\/10.1109\/SC41405.2020.00049","DOI":"10.1109\/SC41405.2020.00049"},{"key":"20666_CR36","unstructured":"Zhuang Y, Zhao H, Zheng L, Li Z, Xing EP, Ho Q, Gonzalez JE, Stoica I, Zhang H (2022) On optimizing the communication of model parallelism"},{"key":"20666_CR37","unstructured":"AMD EPYC\u2122 9684X, General Specifications. https:\/\/www.amd.com\/en\/products\/cpu\/amd-epyc-9684x. Accessed on 18 Aug 2023"},{"key":"20666_CR38","unstructured":"Sovrasov V. Ptflops: a flops counting tool for neural networks in pytorch framework"},{"key":"20666_CR39","unstructured":"Li B, Wang Y, Wang T, Eeckhout L, Yang J, Jaleel A, Tang X (2024) Improving multi-instance GPU efficiency via sub-entry sharing TLB design. arXiv:2404.18361"},{"key":"20666_CR40","doi-asserted-by":"publisher","unstructured":"Velten M, Sch\u00f6ne R, Ilsche T, Hackenberg D (2022) Memory performance of amd epyc rome and intel cascade lake sp server processors. In: Proceedings of the 2022 ACM\/SPEC on International Conference on Performance Engineering. ICPE\u201922. ACM. https:\/\/doi.org\/10.1145\/3489525.3511689","DOI":"10.1145\/3489525.3511689"},{"key":"20666_CR41","unstructured":"Efficient Training on a Single GPU \u2014 huggingface.co. https:\/\/huggingface.co\/docs\/transformers\/v4.20.1\/en\/perf_train_gpu_one#anatomy-of-models-memory. Accessed on 27 Aug 2024"},{"issue":"1","key":"20666_CR42","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1007\/s11063-024-11445-y","volume":"56","author":"M Pa\u0161en","year":"2024","unstructured":"Pa\u0161en M, Bo\u017ea V (2024) Merging of neural networks. Neural Process Lett 56(1):8. https:\/\/doi.org\/10.1007\/s11063-024-11445-y","journal-title":"Neural Process Lett"},{"key":"20666_CR43","unstructured":"Yun D (2024) MeGA: merging multiple independently trained neural networks based on genetic algorithm. arXiv:2406.04607"}],"container-title":["Multimedia Tools and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-025-20666-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11042-025-20666-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11042-025-20666-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,12]],"date-time":"2025-09-12T12:33:01Z","timestamp":1757680381000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11042-025-20666-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,5]]},"references-count":43,"journal-issue":{"issue":"30","published-online":{"date-parts":[[2025,9]]}},"alternative-id":["20666"],"URL":"https:\/\/doi.org\/10.1007\/s11042-025-20666-9","relation":{},"ISSN":["1573-7721"],"issn-type":[{"value":"1573-7721","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,5]]},"assertion":[{"value":"26 October 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 December 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 January 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 February 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}