{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,22]],"date-time":"2025-02-22T00:35:39Z","timestamp":1740184539950,"version":"3.37.3"},"reference-count":62,"publisher":"IOP Publishing","issue":"2","license":[{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T00:00:00Z","timestamp":1685059200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"Austrian Science Fund","award":["I 4670-N"],"award-info":[{"award-number":["I 4670-N"]}]},{"DOI":"10.13039\/100010664","name":"H2020 Future and Emerging Technologies","doi-asserted-by":"crossref","award":["824162"],"award-info":[{"award-number":["824162"]}],"id":[{"id":"10.13039\/100010664","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Neuromorph. Comput. Eng."],"published-print":{"date-parts":[[2023,6,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Mixed-signal and fully digital neuromorphic systems have been of significant interest for deploying spiking neural networks in an energy-efficient manner. However, many of these systems impose constraints in terms of fan-in, memory, or synaptic weight precision that have to be considered during network design and training. In this paper, we present quantized rewiring (Q-rewiring), an algorithm that can train both spiking and non-spiking neural networks while meeting hardware constraints during the entire training process. To demonstrate our approach, we train both feedforward and recurrent neural networks with a combined fan-in\/weight precision limit, a constraint that is, for example, present in the DYNAP-SE mixed-signal analog\/digital neuromorphic processor. Q-rewiring simultaneously performs quantization and rewiring of synapses and synaptic weights through gradient descent updates and projecting the trainable parameters to a constraint-compliant region. Using our algorithm, we find trade-offs between the number of incoming connections to neurons and network performance for a number of common benchmark datasets.<\/jats:p>","DOI":"10.1088\/2634-4386\/accd8f","type":"journal-article","created":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T06:06:26Z","timestamp":1685081186000},"page":"024006","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Quantized rewiring: hardware-aware training of sparse deep neural networks"],"prefix":"10.1088","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3106-7008","authenticated-orcid":true,"given":"Horst","family":"Petschenig","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8724-5507","authenticated-orcid":false,"given":"Robert","family":"Legenstein","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2023,5,26]]},"reference":[{"year":"2020","author":"Anthony","article-title":"Carbontracker: tracking and predicting the carbon footprint of training deep learning models","key":"nceaccd8fbib1"},{"key":"nceaccd8fbib2","doi-asserted-by":"publisher","first-page":"613","DOI":"10.1145\/359576.359579","article-title":"Can programming be liberated from the von Neumann style? A functional style and its algebra of programs","volume":"21","author":"Backus","year":"1978","journal-title":"Commun. ACM"},{"year":"2017","author":"Bellec","article-title":"Deep rewiring: training very sparse deep networks","key":"nceaccd8fbib3"},{"key":"nceaccd8fbib4","article-title":"Long short-term memory and learning-to-learn in networks of spiking neurons","volume":"vol 31","author":"Bellec","year":"2018"},{"key":"nceaccd8fbib5","doi-asserted-by":"publisher","first-page":"3625","DOI":"10.1038\/s41467-020-17236-y","article-title":"A solution to the learning dilemma for recurrent networks of spiking neurons","volume":"11","author":"Bellec","year":"2020","journal-title":"Nat. Commun."},{"key":"nceaccd8fbib6","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1016\/j.carbpol.2013.10.100","article-title":"Neurogrid: a mixed-analog-digital multichip system for large-scale neural simulations","volume":"102","author":"Benjamin","year":"2014","journal-title":"Proc. IEEE"},{"key":"nceaccd8fbib7","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.neunet.2020.09.024","article-title":"Structural plasticity on an accelerated analog neuromorphic hardware system","volume":"133","author":"Billaudelle","year":"2021","journal-title":"Neural Netw."},{"volume":"vol 4","year":"2006","author":"Bishop","key":"nceaccd8fbib8"},{"key":"nceaccd8fbib9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TNNLS.2022.3153985","article-title":"Online spatio-temporal learning in deep neural networks","author":"Bohnstingl","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"nceaccd8fbib10","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2109194119","article-title":"Surrogate gradients for analog neuromorphic computing","volume":"119","author":"Cramer","year":"2022","journal-title":"Proc. Natl Acad. Sci."},{"key":"nceaccd8fbib11","first-page":"pp 702","article-title":"RandAugment: practical automated data augmentation with a reduced search space","author":"Cubuk","year":"2020"},{"key":"nceaccd8fbib12","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MM.2018.112130359","article-title":"Loihi: a neuromorphic manycore processor with on-chip learning","volume":"38","author":"Davies","year":"2018","journal-title":"IEEE Micro"},{"key":"nceaccd8fbib13","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1109\/JPROC.2021.3067593","article-title":"Advancing neuromorphic computing with Loihi: a survey of results and outlook","volume":"109","author":"Davies","year":"2021","journal-title":"Proc. IEEE"},{"key":"nceaccd8fbib14","doi-asserted-by":"publisher","first-page":"11441","DOI":"10.1073\/pnas.1604850113","article-title":"Convolutional networks for fast, energy-efficient neuromorphic computing","volume":"113","author":"Esser","year":"2016","journal-title":"Proc. Natl Acad. Sci."},{"year":"2019","author":"Frankle","article-title":"The lottery ticket hypothesis: finding sparse, trainable neural networks","key":"nceaccd8fbib15"},{"key":"nceaccd8fbib16","first-page":"pp 1","article-title":"ReckOn: a 28 nm sub-mm2 task-agnostic spiking recurrent neural network processor enabling on-chip learning over second-long timescales","volume":"vol 65","author":"Frenkel","year":"2022"},{"key":"nceaccd8fbib17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TNNLS.2022.3180209","article-title":"Spartus: a 9.4 TOp\/s FPGA-based LSTM accelerator exploiting spatio-temporal sparsity","author":"Gao","year":"2022","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"nceaccd8fbib18","doi-asserted-by":"publisher","first-page":"823","DOI":"10.1038\/s42256-021-00388-x","article-title":"Fast and energy-efficient neuromorphic deep learning with first-spike times","volume":"3","author":"G\u00f6ltz","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"nceaccd8fbib19","article-title":"Learning both weights and connections for efficient neural network","volume":"vol 28","author":"Han","year":"2015"},{"year":"2015","author":"Kaiming","article-title":"Delving deep into rectifiers: surpassing human-level performance on ImageNet classification","key":"nceaccd8fbib20"},{"key":"nceaccd8fbib21","first-page":"pp 770","article-title":"Deep residual learning for image recognition","author":"Kaiming","year":"2016"},{"key":"nceaccd8fbib22","first-page":"pp 16783","article-title":"PixMix: dreamlike pictures comprehensively improve safety measures","author":"Hendrycks","year":"2022"},{"key":"nceaccd8fbib23","doi-asserted-by":"publisher","first-page":"647","DOI":"10.1038\/nrn2699","article-title":"Experience-dependent structural synaptic plasticity in the mammalian brain","volume":"10","author":"Holtmaat","year":"2009","journal-title":"Nat. Rev. Neurosci."},{"key":"nceaccd8fbib24","first-page":"6869","article-title":"Quantized neural networks: training neural networks with low precision weights and activations","volume":"18","author":"Hubara","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"nceaccd8fbib25","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR42600.2020.00222","article-title":"AdaBits: neural network quantization with adaptive bit-widths","author":"Jin","year":"2020"},{"key":"nceaccd8fbib26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pcbi.1004485","article-title":"Network plasticity as Bayesian inference","volume":"11","author":"Kappel","year":"2015","journal-title":"PLoS Comput. Biol."},{"key":"nceaccd8fbib27","article-title":"Synaptic sampling: a Bayesian approach to neural network plasticity and rewiring","volume":"vol 28","author":"Kappel","year":"2015"},{"year":"2009","author":"Krizhevsky","article-title":"Learning multiple layers of features from tiny images","key":"nceaccd8fbib28"},{"key":"nceaccd8fbib29","article-title":"ImageNet classification with deep convolutional neural networks","volume":"vol 25","author":"Krizhevsky","year":"2012"},{"year":"2015","author":"Le","article-title":"A simple way to initialize recurrent networks of rectified linear units","key":"nceaccd8fbib30"},{"key":"nceaccd8fbib31","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"nceaccd8fbib32","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/j.neucom.2021.07.045","article-title":"Pruning and quantization for deep neural network acceleration: a survey","volume":"461","author":"Liang","year":"2021","journal-title":"Neurocomputing"},{"key":"nceaccd8fbib33","doi-asserted-by":"publisher","first-page":"840","DOI":"10.3389\/fnins.2018.00840","article-title":"Memory-efficient deep learning on a SpiNNaker 2 prototype","volume":"12","author":"Liu","year":"2018","journal-title":"Front. Neurosci."},{"year":"2019","author":"Liu","article-title":"Rethinking the value of network pruning","key":"nceaccd8fbib34"},{"year":"2016","author":"Loshchilov","article-title":"SGDR: stochastic gradient descent with warm restarts","key":"nceaccd8fbib35"},{"year":"2001","author":"Maass","key":"nceaccd8fbib36"},{"key":"nceaccd8fbib37","doi-asserted-by":"crossref","DOI":"10.21437\/Interspeech.2020-1058","article-title":"MatchboxNet: 1D time-channel separable convolutional neural network architecture for speech commands recognition","author":"Majumdar","year":"2020"},{"key":"nceaccd8fbib38","doi-asserted-by":"publisher","first-page":"668","DOI":"10.1126\/science.1254642","article-title":"A million spiking-neuron integrated circuit with a scalable communication network and interface","volume":"345","author":"Merolla","year":"2014","journal-title":"Science"},{"key":"nceaccd8fbib39","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1109\/TBCAS.2017.2759700","article-title":"A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs)","volume":"12","author":"Moradi","year":"2017","journal-title":"IEEE Trans. Biomed. Circuits Syst."},{"key":"nceaccd8fbib40","doi-asserted-by":"publisher","first-page":"1672","DOI":"10.1038\/nn.4403","article-title":"History-dependent variability in population dynamics during evidence accumulation in cortex","volume":"19","author":"Morcos","year":"2016","journal-title":"Nat. Neurosci."},{"key":"nceaccd8fbib41","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1109\/JPROC.2018.2881432","article-title":"Braindrop: a mixed-signal neuromorphic architecture with a dynamical systems-based programming model","volume":"107","author":"Neckar","year":"2018","journal-title":"Proc. IEEE"},{"year":"1999","author":"Nocedal","key":"nceaccd8fbib42"},{"key":"nceaccd8fbib43","article-title":"Direct feedback alignment provides learning in deep neural networks","volume":"vol 29","author":"N\u00f8kland","year":"2016"},{"key":"nceaccd8fbib44","doi-asserted-by":"publisher","first-page":"1943","DOI":"10.1109\/JSSC.2013.2259038","article-title":"SpiNNaker: a 1-w 18-core system-on-chip for massively-parallel neural network simulation","volume":"48","author":"Painkras","year":"2013","journal-title":"IEEE J. Solid-State Circuits"},{"key":"nceaccd8fbib45","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1038\/s41586-019-1424-8","article-title":"Towards artificial general intelligence with hybrid Tianjic chip architecture","volume":"572","author":"Pei","year":"2019","journal-title":"Nature"},{"key":"nceaccd8fbib46","first-page":"pp 97","article-title":"Low-activity supervised convolutional spiking neural networks applied to speech commands recognition","author":"Pellegrini","year":"2021"},{"key":"nceaccd8fbib47","doi-asserted-by":"publisher","first-page":"141","DOI":"10.3389\/fnins.2015.00141","article-title":"A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses","volume":"9","author":"Qiao","year":"2015","journal-title":"Front. Neurosci."},{"key":"nceaccd8fbib48","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1038\/323533a0","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart","year":"1986","journal-title":"Nature"},{"key":"nceaccd8fbib49","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.65459","article-title":"Spike frequency adaptation supports network computations on temporally dispersed information","volume":"10","author":"Salaj","year":"2021","journal-title":"eLife"},{"key":"nceaccd8fbib50","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1016\/j.neunet.2014.09.003","article-title":"Deep learning in neural networks: an overview","volume":"61","author":"Schmidhuber","year":"2015","journal-title":"Neural Netw."},{"key":"nceaccd8fbib51","first-page":"pp 2227","article-title":"Neuromorphic hardware in the loop: training a deep spiking network on the brainscales wafer-scale system","author":"Schmitt","year":"2017"},{"key":"nceaccd8fbib52","doi-asserted-by":"crossref","DOI":"10.21437\/Interspeech.2019-1873","article-title":"wav2vec: unsupervised pre-training for speech recognition","author":"Schneider","year":"2019"},{"key":"nceaccd8fbib53","first-page":"pp 31.1","article-title":"Data-free parameter pruning for deep neural networks","author":"Srinivas","year":"2015"},{"key":"nceaccd8fbib54","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2016.308","article-title":"Rethinking the inception architecture for computer vision","author":"Szegedy","year":"2016"},{"key":"nceaccd8fbib55","first-page":"pp 6105","article-title":"EfficientNet: rethinking model scaling for convolutional neural networks","author":"Tan","year":"2019"},{"key":"nceaccd8fbib56","first-page":"pp 5741","article-title":"Bayesian bits: unifying quantization and pruning","volume":"vol 33","author":"Van Baalen","year":"2020"},{"key":"nceaccd8fbib57","first-page":"pp 498","article-title":"33.1 a 74 TMACS\/W CMOS-RRAM neurosynaptic core with dynamically reconfigurable dataflow and in-situ transposable weights for probabilistic graphical models","author":"Wan","year":"2020"},{"year":"2018","author":"Warden","article-title":"Speech commands: a dataset for limited-vocabulary speech recognition","key":"nceaccd8fbib58"},{"key":"nceaccd8fbib59","first-page":"pp 681","article-title":"Bayesian learning via stochastic gradient Langevin dynamics","author":"Welling","year":"2011"},{"key":"nceaccd8fbib60","doi-asserted-by":"publisher","first-page":"1550","DOI":"10.1109\/5.58337","article-title":"Backpropagation through time: what it does and how to do it","volume":"78","author":"Werbos","year":"1990","journal-title":"Proc. IEEE"},{"volume":"vol 1","year":"1994","author":"Werbos","key":"nceaccd8fbib61"},{"key":"nceaccd8fbib62","first-page":"pp T86","article-title":"RRAM-based spiking nonvolatile computing-in-memory processing engine with precision-configurable in situ nonlinear activation","author":"Yan","year":"2019"}],"container-title":["Neuromorphic Computing and Engineering"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/accd8f","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/accd8f\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/accd8f\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/accd8f\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T06:06:31Z","timestamp":1685081191000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2634-4386\/accd8f"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,26]]},"references-count":62,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,5,26]]},"published-print":{"date-parts":[[2023,6,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2634-4386\/accd8f","relation":{},"ISSN":["2634-4386"],"issn-type":[{"type":"electronic","value":"2634-4386"}],"subject":[],"published":{"date-parts":[[2023,5,26]]},"assertion":[{"value":"Quantized rewiring: hardware-aware training of sparse deep neural networks","name":"article_title","label":"Article Title"},{"value":"Neuromorphic Computing and Engineering","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2023 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2022-10-07","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-02-02","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2023-05-26","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}