{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T05:20:13Z","timestamp":1783056013418,"version":"3.54.6"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T00:00:00Z","timestamp":1779926400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T00:00:00Z","timestamp":1779926400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput &amp; Applic"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Model compression is critical for deploying deep learning models on resource-constrained devices. We introduce a novel method enhancing knowledge distillation with Integrated Gradients (IG) as a data augmentation strategy. Our approach overlays IG maps onto input images during training, providing student models with deeper insights into teacher models\u2019 decision-making processes. Extensive evaluation on CIFAR-10 demonstrates that our IG-augmented knowledge distillation achieves 92.6% testing accuracy with a 4.1x compression factor\u2013a significant 1.1 percentage point improvement (p &lt; 0.001) over non-distilled models (91.5%). This compression reduces inference time from 140ms to 13ms. Our method precomputes IG maps before training, transforming substantial runtime costs into a one-time preprocessing step. We validate our approach through systematic ablation studies with attention transfer, comprehensive compression factor analysis (2.2\n                    <jats:inline-formula>\n                      <jats:tex-math>$$\\times$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    \u20131122\n                    <jats:inline-formula>\n                      <jats:tex-math>$$\\times$$<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    ), Monte Carlo simulations, and cross-dataset evaluation on ImageNet subsets, demonstrating that IG-based knowledge distillation consistently outperforms conventional approaches. Our results establish this framework as a viable compression technique for real-world deployment on edge devices while maintaining competitive accuracy.\n                  <\/jats:p>","DOI":"10.1007\/s00521-026-12137-5","type":"journal-article","created":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T02:47:28Z","timestamp":1779936448000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Model compression using knowledge distillation with integrated gradients"],"prefix":"10.1007","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-4392-6927","authenticated-orcid":false,"given":"David E.","family":"Hernandez","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5587-7828","authenticated-orcid":false,"given":"Jose Ramon","family":"Chang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4867-6707","authenticated-orcid":false,"given":"Torbj\u00f6rn E. M.","family":"Nordling","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,5,28]]},"reference":[{"key":"12137_CR1","unstructured":"Ashok A, Rhinehart N, Beainy F, et al (2017) N2n learning: Network to network compression via policy gradient reinforcement learning. arXiv preprint arXiv:1709.06030"},{"issue":"7","key":"12137_CR2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach S, Binder A, Montavon G et al (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 10(7):e0130140","journal-title":"PLoS One"},{"key":"12137_CR3","doi-asserted-by":"crossref","unstructured":"Bengio Y, Louradour J, Collobert R, et al (2009) Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning, pp 41\u201348","DOI":"10.1145\/1553374.1553380"},{"key":"12137_CR4","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3358205","volume":"18","author":"K Bhardwaj","year":"2019","unstructured":"Bhardwaj K, Lin CY, Sartor A et al (2019) Memory- and communication-aware model compression for distributed deep learning inference on iot. ACM Transactions on Embedded Computing Systems 18:1\u201322. https:\/\/doi.org\/10.1145\/3358205","journal-title":"ACM Transactions on Embedded Computing Systems"},{"key":"12137_CR5","doi-asserted-by":"crossref","unstructured":"Chen H, Wang Y, Xu C, et al (2019) Data-free learning of student networks. In: Proceedings of the IEEE\/CVF international conference on computer vision, pp 3514\u20133522","DOI":"10.1109\/ICCV.2019.00361"},{"key":"12137_CR6","doi-asserted-by":"publisher","unstructured":"Chen L, Varoquaux G (2024) What is the role of small models in the llm era: A survey. arXiv preprint arXiv:2409.06857https:\/\/doi.org\/10.48550\/arXiv.2409.06857","DOI":"10.48550\/arXiv.2409.06857"},{"key":"12137_CR7","unstructured":"Cheng Y, Wang D, Zhou P, et al (2017) A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282"},{"key":"12137_CR8","doi-asserted-by":"crossref","unstructured":"Choi Y, Choi J, El-Khamy M, et al (2020) Data-free network quantization with adversarial knowledge distillation. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp 710\u2013711","DOI":"10.1109\/CVPRW50498.2020.00363"},{"issue":"4","key":"12137_CR9","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1109\/JPROC.2020.2976475","volume":"108","author":"L Deng","year":"2020","unstructured":"Deng L, Li G, Han S et al (2020) Model compression and hardware acceleration for neural networks: a comprehensive survey. Proc IEEE 108(4):485\u2013532","journal-title":"Proc IEEE"},{"issue":"6","key":"12137_CR10","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","volume":"129","author":"J Gou","year":"2021","unstructured":"Gou J, Yu B, Maybank SJ et al (2021) Knowledge distillation: a survey. Int J Comput Vis 129(6):1789\u20131819","journal-title":"Int J Comput Vis"},{"issue":"5","key":"12137_CR11","doi-asserted-by":"publisher","first-page":"7099","DOI":"10.1109\/TII.2022.3209672","volume":"19","author":"J Gou","year":"2022","unstructured":"Gou J, Sun L, Yu B et al (2022) Multilevel attention-based sample correlations for knowledge distillation. IEEE Trans Industr Inf 19(5):7099\u20137109","journal-title":"IEEE Trans Industr Inf"},{"issue":"2","key":"12137_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3568679","volume":"20","author":"J Gou","year":"2023","unstructured":"Gou J, Sun L, Yu B et al (2023) Hierarchical multi-attention transfer for knowledge distillation. ACM Trans Multimed Comput Commun Appl 20(2):1\u201320","journal-title":"ACM Trans Multimed Comput Commun Appl"},{"key":"12137_CR13","unstructured":"Han S, Mao H, Dally WJ (2015) Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149"},{"key":"12137_CR14","doi-asserted-by":"crossref","unstructured":"Hao Z, Guo J, Han K, et al (2023) Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale. arXiv preprint arXiv:2305.15781","DOI":"10.52202\/075280-0444"},{"key":"12137_CR15","unstructured":"Hinton G, Vinyals O, Dean J (2015) Distilling the Knowledge in a Neural Network. http:\/\/arxiv.org\/abs\/1503.02531"},{"key":"12137_CR16","unstructured":"Howard AG, Zhu M, Chen B, et al (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861"},{"key":"12137_CR17","unstructured":"Hubara I, Courbariaux M, Soudry D, et al (2016) Binarized neural networks. Advances in neural information processing systems 29"},{"key":"12137_CR18","doi-asserted-by":"crossref","unstructured":"Idelbayev Y, Carreira-Perpin\u00e1n MA (2020) Low-rank compression of neural nets: Learning the rank of each layer. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp 8049\u20138059","DOI":"10.1109\/CVPR42600.2020.00807"},{"key":"12137_CR19","doi-asserted-by":"crossref","unstructured":"Jaderberg M, Vedaldi A, Zisserman A (2014) Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866","DOI":"10.5244\/C.28.88"},{"key":"12137_CR20","unstructured":"Krishnamoorthi R (2018) Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342"},{"key":"12137_CR21","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1016\/j.neucom.2021.04.141","volume":"485","author":"D Liu","year":"2022","unstructured":"Liu D, Kong H, Luo X et al (2022) Bringing ai to edge: from deep learning\u2019s perspective. Neurocomputing 485:297\u2013320","journal-title":"Neurocomputing"},{"key":"12137_CR22","unstructured":"Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS) Https:\/\/arxiv.org\/abs\/1705.07874"},{"key":"12137_CR23","doi-asserted-by":"crossref","unstructured":"Markidis S, Der Chien SW, Laure E, et al (2018) Nvidia tensor core programmability, performance & precision. In: 2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW), IEEE, pp 522\u2013531","DOI":"10.1109\/IPDPSW.2018.00091"},{"key":"12137_CR24","unstructured":"Molchanov P, Tyree S, Karras T, et al (2016) Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440"},{"issue":"11","key":"12137_CR25","first-page":"1","volume":"54","author":"Y Pan","year":"2018","unstructured":"Pan Y, Ouyang P, Zhao Y et al (2018) A multilevel cell stt-mram-based computing in-memory accelerator for binary convolutional neural network. IEEE Trans Magn 54(11):1\u20135","journal-title":"IEEE Trans Magn"},{"key":"12137_CR26","doi-asserted-by":"crossref","unstructured":"Park W, Kim D, Lu Y, et al (2019) Relational knowledge distillation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 3967\u20133976","DOI":"10.1109\/CVPR.2019.00409"},{"key":"12137_CR27","doi-asserted-by":"crossref","unstructured":"Rangwani H, Mondal P, Mishra M, et al (2024) Deit-lt: Distillation strikes back for vision transformer training on long-tailed datasets. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp 23396\u201323406","DOI":"10.1109\/CVPR52733.2024.02208"},{"key":"12137_CR28","doi-asserted-by":"crossref","unstructured":"Ribeiro MT, Singh S, Guestrin C (2016) \u201cwhy should i trust you?\u201d: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining Https:\/\/arxiv.org\/abs\/1602.04938","DOI":"10.1145\/2939672.2939778"},{"key":"12137_CR29","unstructured":"Romero A, Ballas N, Kahou SE, et al (2014) FitNets: Hints for Thin Deep Nets. arXiv preprint arXiv:1412.6550"},{"key":"12137_CR30","doi-asserted-by":"crossref","unstructured":"Selvaraju RR, Cogswell M, Das A, et al (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618\u2013626","DOI":"10.1109\/ICCV.2017.74"},{"key":"12137_CR31","unstructured":"Simonyan K, Vedaldi A, Zisserman A (2013) Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034"},{"key":"12137_CR32","unstructured":"Smilkov D, Thorat N, Kim B, et al (2017) Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825"},{"key":"12137_CR33","doi-asserted-by":"crossref","unstructured":"Sun S, Ren W, Li J, et al (2024) Logit standardization in knowledge distillation. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition, pp 15731\u201315740","DOI":"10.1109\/CVPR52733.2024.01489"},{"key":"12137_CR34","unstructured":"Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning, PMLR, pp 3319\u20133328"},{"issue":"12","key":"12137_CR35","doi-asserted-by":"publisher","first-page":"2295","DOI":"10.1109\/JPROC.2017.2761740","volume":"105","author":"V Sze","year":"2017","unstructured":"Sze V, Chen YH, Yang TJ et al (2017) Efficient processing of deep neural networks: A tutorial and survey. Proc IEEE 105(12):2295\u20132329","journal-title":"Proc IEEE"},{"key":"12137_CR36","doi-asserted-by":"crossref","unstructured":"Szegedy C, Ioffe S, Vanhoucke V, et al (2017) Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. In: Thirty-First AAAI Conference on Artificial Intelligence","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"12137_CR37","doi-asserted-by":"crossref","unstructured":"Wang H, Lohit S, Jones MN, et al (2022) What makes a \u201cgood\u201d data augmentation in knowledge distillation - a statistical perspective. In: Koyejo S, Mohamed S, Agarwal A, et al (eds) Advances in Neural Information Processing Systems, vol\u00a035. Curran Associates, Inc., pp 13456\u201313469, https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/ 57b53238ff22bc0dc62de08f53eb5de2-Paper-Conference.pdf","DOI":"10.52202\/068431-0978"},{"key":"12137_CR38","doi-asserted-by":"crossref","unstructured":"Wang T, Wang K, Cai H, et al (2020a) Apq: Joint search for network architecture, pruning and quantization policy. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp 2078\u20132087","DOI":"10.1109\/CVPR42600.2020.00215"},{"key":"12137_CR39","unstructured":"Wang W, Chen W, Luo Y, et al (2024) Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.09748"},{"key":"12137_CR40","doi-asserted-by":"crossref","unstructured":"Wang Y, Zhang X, Xie L, et al (2020b) Pruning from scratch. In: Proceedings of the AAAI conference on artificial intelligence, pp 12273\u201312280","DOI":"10.1609\/aaai.v34i07.6910"},{"key":"12137_CR41","doi-asserted-by":"crossref","unstructured":"Wu S, Chen H, Quan X, et al (2023) Ad-kd: Attribution-driven knowledge distillation for language model compression. arXiv preprint arXiv:2305.10010","DOI":"10.18653\/v1\/2023.acl-long.471"},{"key":"12137_CR42","doi-asserted-by":"crossref","unstructured":"Xia M, Zhong Z, Chen D (2022) Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408","DOI":"10.18653\/v1\/2022.acl-long.107"},{"key":"12137_CR43","unstructured":"Zagoruyko S, Komodakis N (2016) Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer. arXiv preprint arXiv:1612.03928http:\/\/arxiv.org\/abs\/1612.03928"},{"key":"12137_CR44","unstructured":"Zhou Y, Lyu K, Rawat AS, et al (2023) Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-12137-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-026-12137-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-026-12137-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T04:55:14Z","timestamp":1783054514000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-026-12137-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,28]]},"references-count":44,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["12137"],"URL":"https:\/\/doi.org\/10.1007\/s00521-026-12137-5","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"value":"0941-0643","type":"print"},{"value":"1433-3058","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,28]]},"assertion":[{"value":"14 April 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 April 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 May 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no financial or non-financial interests that are directly or indirectly related to the work submitted for publication.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"434"}}