{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T21:32:35Z","timestamp":1742938355450,"version":"3.40.3"},"publisher-location":"Singapore","reference-count":27,"publisher":"Springer Nature Singapore","isbn-type":[{"type":"print","value":"9789819609659"},{"type":"electronic","value":"9789819609666"}],"license":[{"start":{"date-parts":[[2024,12,7]],"date-time":"2024-12-07T00:00:00Z","timestamp":1733529600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2024,12,7]],"date-time":"2024-12-07T00:00:00Z","timestamp":1733529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"DOI":"10.1007\/978-981-96-0966-6_24","type":"book-chapter","created":{"date-parts":[[2024,12,6]],"date-time":"2024-12-06T08:40:34Z","timestamp":1733474434000},"page":"402-418","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["ReLUifying Smooth Functions: Low-Cost Knowledge Distillation to\u00a0Obtain High-Performance ReLU Networks"],"prefix":"10.1007","author":[{"given":"Soosung","family":"Kim","sequence":"first","affiliation":[]},{"given":"Yeonhong","family":"Park","sequence":"additional","affiliation":[]},{"given":"Hyunseung","family":"Lee","sequence":"additional","affiliation":[]},{"given":"Sungchan","family":"Yi","sequence":"additional","affiliation":[]},{"given":"Jae W.","family":"Lee","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,12,7]]},"reference":[{"key":"24_CR1","unstructured":"Prajit Ramachandran, Barret Zoph, and Quoc\u00a0V. Le. Searching for activation functions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018"},{"key":"24_CR2","unstructured":"Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint\u00a0arXiv:1606.08415, 2016"},{"key":"24_CR3","unstructured":"Diganta Misra. Mish: A self regularized non-monotonic neural activation function. arXiv preprint\u00a0arXiv:1908.08681, 2019"},{"key":"24_CR4","unstructured":"Andrew\u00a0L. Maas, Awni\u00a0Y. Hannun, and Andrew\u00a0Y. Ng. Rectifier nonlinearities improve neural network acoustic models. in ICML Workshop on Deep Learning for Audo, Speech, and Language Processing (WDLASL), 2013"},{"key":"24_CR5","unstructured":"ARM Ethos-U65 NPU. https:\/\/developer.arm.com\/documentation\/102023\/0000, 2020"},{"key":"24_CR6","unstructured":"Coral dev board. https:\/\/coral.ai\/docs\/dev-board\/datasheet\/, 2020"},{"key":"24_CR7","unstructured":"Vitis ai. https:\/\/docs.xilinx.com\/r\/en-US\/ug1414-vitis-ai, 2020"},{"key":"24_CR8","unstructured":"Sehoon Kim, Amir Gholami, Zhewei Yao, Michael\u00a0W. Mahoney, and Kurt Keutzer. I-BERT: integer-only BERT quantization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5506\u20135518. PMLR, 2021"},{"key":"24_CR9","doi-asserted-by":"crossref","unstructured":"Jun-Woo Jang, Sehwan Lee, Dongyoung Kim, Hyunsun Park, Ali\u00a0Shafiee Ardestani, Yeongjae Choi, Channoh Kim, Yoojin Kim, Hyeongseok Yu, Hamzah Abdel-Aziz, Jun-Seok Park, Heonsoo Lee, Dongwoo Lee, Myeong\u00a0Woo Kim, Hanwoong Jung, Heewoo Nam, Dongguen Lim, Seungwon Lee, Joon-Ho Song, Suknam Kwon, Joseph Hassoun, SukHwan Lim, and Changkyu Choi. Sparsity-aware and re-configurable NPU architecture for samsung flagship mobile soc. In 48th ACM\/IEEE Annual International Symposium on Computer Architecture, ISCA 2021, Valencia, Spain, June 14-18, 2021, pages 15\u201328. IEEE, 2021","DOI":"10.1109\/ISCA52012.2021.00011"},{"key":"24_CR10","unstructured":"Xiaohu Tang, Shihao Han, Li\u00a0Lyna Zhang, Ting Cao, and Yunxin Liu. To bridge neural network design and real-world performance: A behaviour study for neural networks. In Alex Smola, Alex Dimakis, and Ion Stoica, editors, Proceedings of Machine Learning and Systems 2021, MLSys 2021, virtual, April 5-9, 2021. mlsys.org, 2021"},{"key":"24_CR11","doi-asserted-by":"crossref","unstructured":"Pavan Kumar\u00a0Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 7907\u20137917. IEEE, 2023","DOI":"10.1109\/CVPR52729.2023.00764"},{"key":"24_CR12","unstructured":"Neo\u00a0Wei Ming, Zhehui Wang, Cheng Liu, Rick Siow\u00a0Mong Goh, and Tao Luo. MA-BERT: towards matrix arithmetic-only BERT inference by eliminating complex non-linear functions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023"},{"key":"24_CR13","doi-asserted-by":"crossref","unstructured":"Jing Chen and Xue Liu. A high-performance deeply pipelined architecture for elementary transcendental function evaluation. In 2017 IEEE International Conference on Computer Design (ICCD), pages 209\u2013216, 2017","DOI":"10.1109\/ICCD.2017.39"},{"key":"24_CR14","unstructured":"Joonsang Yu, Junki Park, Seongmin Park, Minsoo Kim, Sihwa Lee, Dong\u00a0Hyun Lee, and Jungwook Choi. Nn-lut: Neural approximation of non-linear operations for efficient transformer inference. In Rob Oshana, editor, Proceedings of the 59th ACM\/IEEE Design Automation Conference, DAC \u201922, page 577-582, New York, NY, USA, 2022. Association for Computing Machinery"},{"key":"24_CR15","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1026\u20131034. IEEE Computer Society, 2015"},{"key":"24_CR16","doi-asserted-by":"crossref","unstructured":"Jun-Seok Park, Jun-Woo Jang, Heonsoo Lee, Dongwoo Lee, Sehwan Lee, Hanwoong Jung, Seungwon Lee, Suknam Kwon, Kyung-Ah Jeong, Joon-Ho Song, SukHwan Lim, and Inyup Kang. 9.5 A 6k-mac feature-map-sparsity-aware neural processing unit in 5nm flagship mobile soc. In IEEE International Solid-State Circuits Conference, ISSCC 2021, San Francisco, CA, USA, February 13-22, 2021, pages 152\u2013154. IEEE, 2021","DOI":"10.1109\/ISSCC42613.2021.9365928"},{"key":"24_CR17","unstructured":"Linfeng Zhang, Yukang Shi, Zuoqiang Shi, Kaisheng Ma, and Chenglong Bao. Task-oriented feature distillation. In Hugo Larochelle, Marc\u2019Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020"},{"key":"24_CR18","unstructured":"Jonathan Frankle, Gintare\u00a0Karolina Dziugaite, Daniel\u00a0M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3259\u20133269. PMLR, 2020"},{"key":"24_CR19","unstructured":"Leslie\u00a0N. Smith and Nicholay Topin. Super-convergence: Very fast training of residual networks using large learning rates. arXiv preprint\u00a0arXiv:1708.07120 , 2017"},{"key":"24_CR20","unstructured":"Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander\u00a0A. Alemi, and Andrew\u00a0Gordon Wilson. Does knowledge distillation really work? In Marc\u2019Aurelio Ranzato, Alina Beygelzimer, Yann\u00a0N. Dauphin, Percy Liang, and Jennifer\u00a0Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 6906\u20136919, 2021"},{"key":"24_CR21","doi-asserted-by":"crossref","unstructured":"Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu\u00a0Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In Shai Avidan, Gabriel\u00a0J. Brostow, Moustapha Ciss\u00e9, Giovanni\u00a0Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXI, volume 13681 of Lecture Notes in Computer Science, pages 68\u201385. Springer, 2022","DOI":"10.1007\/978-3-031-19803-8_5"},{"key":"24_CR22","unstructured":"Cream. https:\/\/github.com\/microsoft\/Cream"},{"key":"24_CR23","doi-asserted-by":"crossref","unstructured":"Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin\u00a0Young Choi. A comprehensive overhaul of feature distillation. In 2019 IEEE\/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 1921\u20131930. IEEE, 2019","DOI":"10.1109\/ICCV.2019.00201"},{"key":"24_CR24","unstructured":"Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5008\u20135017. Computer Vision Foundation \/ IEEE, 2021"},{"key":"24_CR25","unstructured":"Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017"},{"key":"24_CR26","doi-asserted-by":"crossref","unstructured":"Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11923\u201311932. IEEE, 2022","DOI":"10.1109\/CVPR52688.2022.01163"},{"key":"24_CR27","doi-asserted-by":"crossref","unstructured":"Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11943\u201311952. IEEE, 2022","DOI":"10.1109\/CVPR52688.2022.01165"}],"container-title":["Lecture Notes in Computer Science","Computer Vision \u2013 ACCV 2024"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-981-96-0966-6_24","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,6]],"date-time":"2024-12-06T09:09:35Z","timestamp":1733476175000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-981-96-0966-6_24"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,7]]},"ISBN":["9789819609659","9789819609666"],"references-count":27,"URL":"https:\/\/doi.org\/10.1007\/978-981-96-0966-6_24","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2024,12,7]]},"assertion":[{"value":"7 December 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of Interests"}},{"value":"ACCV","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Asian Conference on Computer Vision","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Hanoi","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Vietnam","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"8 December 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"12 December 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"17","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"accv2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}