{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T18:14:07Z","timestamp":1779905647106,"version":"3.53.1"},"reference-count":63,"publisher":"IOP Publishing","issue":"4","license":[{"start":{"date-parts":[[2021,7,16]],"date-time":"2021-07-16T00:00:00Z","timestamp":1626393600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,7,16]],"date-time":"2021-07-16T00:00:00Z","timestamp":1626393600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2021,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on field-programmable gate arrays (FPGAs). By extending the <jats:monospace>hls4ml<\/jats:monospace> library, we demonstrate an inference latency of 5\u2009<jats:italic>\u00b5<\/jats:italic>s using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detectors. In particular, we discuss pruning and quantization-aware training, and demonstrate how resource utilization can be significantly reduced with little to no loss in model accuracy. We show that the FPGA critical resource consumption can be reduced by 97% with zero loss in model accuracy, and by 99% when tolerating a 6% accuracy degradation.<\/jats:p>","DOI":"10.1088\/2632-2153\/ac0ea1","type":"journal-article","created":{"date-parts":[[2021,6,25]],"date-time":"2021-06-25T22:15:15Z","timestamp":1624659315000},"page":"045015","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":124,"title":["Fast convolutional neural networks on FPGAs with hls4ml"],"prefix":"10.1088","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7671-243X","authenticated-orcid":false,"given":"Thea","family":"Aarrestad","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vladimir","family":"Loncar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicol\u00f2","family":"Ghielmetti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maurizio","family":"Pierini","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sioni","family":"Summers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0055-2935","authenticated-orcid":false,"given":"Jennifer","family":"Ngadiuba","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoffer","family":"Petersson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hampus","family":"Linander","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yutaro","family":"Iiyama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Giuseppe","family":"Di Guglielmo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5076-7096","authenticated-orcid":false,"given":"Javier","family":"Duarte","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8189-3741","authenticated-orcid":false,"given":"Philip","family":"Harris","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dylan","family":"Rankin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sergo","family":"Jindariani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin","family":"Pedro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nhan","family":"Tran","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mia","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edward","family":"Kreinar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhenbin","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Duc","family":"Hoang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2021,7,16]]},"reference":[{"key":"mlstac0ea1bib1","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/13\/07\/P07027","article-title":"Fast inference of deep neural networks in FPGAs for particle physics","volume":"13","author":"Duarte","year":"2018","journal-title":"J. Instrum."},{"key":"mlstac0ea1bib2","article-title":"fastmachinelearning\/hls4ml: aster","author":"Loncar","year":"2020"},{"key":"mlstac0ea1bib3","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/15\/10\/P10004","article-title":"Operation of the ATLAS trigger system in Run 2","volume":"15","author":"","year":"2020","journal-title":"J. Instrum."},{"key":"mlstac0ea1bib4","article-title":"Technical design report for the Phase-II upgrade of the ATLAS TDAQ system","author":"","year":"2017"},{"key":"mlstac0ea1bib5","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/15\/10\/P10017","article-title":"Performance of the CMS Level-1 trigger in proton-proton collisions at s=13 TeV","volume":"15","author":"","year":"2020","journal-title":"J. Instrum."},{"key":"mlstac0ea1bib6","article-title":"The phase-2 upgrade of the CMS level-1 trigger","author":"","year":"2020"},{"key":"mlstac0ea1bib7","article-title":"Vivado design suite user guide: high-level synthesis","author":"","year":"2020"},{"key":"mlstac0ea1bib8","doi-asserted-by":"crossref","article-title":"Compressing deep neural networks on FPGAs to binary and ternary precision with hls4ml","author":"Ngadiuba","DOI":"10.1088\/2632-2153\/aba042"},{"key":"mlstac0ea1bib9","doi-asserted-by":"publisher","DOI":"10.1088\/1748-0221\/15\/05\/P05026","article-title":"Fast inference of boosted decision trees in FPGAs for particle physics","volume":"15","author":"Summers","year":"2020","journal-title":"J. Instrum."},{"key":"mlstac0ea1bib10","doi-asserted-by":"publisher","first-page":"44","DOI":"10.3389\/fdata.2020.598927","article-title":"Distance-weighted graph neural networks on FPGAs for real-time particle reconstruction in high energy physics","volume":"3","author":"Iiyama","year":"2021","journal-title":"Front. Big Data"},{"key":"mlstac0ea1bib11","article-title":"Accelerated charged particle tracking with graph neural networks on FPGAs","volume":"vol 12","author":"Heintz","year":"2020"},{"key":"mlstac0ea1bib12","article-title":"TensorFlow: large-scale machine learning on heterogeneous systems","author":"Abadi","year":"2015"},{"key":"mlstac0ea1bib13","article-title":"Keras","author":"Chollet","year":"2015"},{"key":"mlstac0ea1bib14","first-page":"p 8024","article-title":"PyTorch: an imperative style, high-performance deep learning library","author":"Paszke","year":"2019"},{"key":"mlstac0ea1bib15","article-title":"ONNX","author":"","year":"2017"},{"key":"mlstac0ea1bib16","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-021-00356-5","article-title":"Automatic deep heterogeneous quantization of deep neural networks for ultra low-area, low-latency inference on the edge at particle colliders","author":"Coelho","year":"2021","journal-title":"Nat. Mach. Intell"},{"key":"mlstac0ea1bib17","article-title":"Reading digits in natural images with unsupervised feature learning","author":"Netzer","year":"2011"},{"key":"mlstac0ea1bib18","article-title":"Convolutional neural networks for track reconstruction on FPGAs","author":"Boser","year":"2017"},{"key":"mlstac0ea1bib19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3186332","article-title":"Toolflows for mapping convolutional neural networks on FPGAs: a survey and future directions","volume":"51","author":"Venieris","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"mlstac0ea1bib20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3289185","article-title":"A survey of FPGA-based neural network inference accelerators","volume":"12","author":"Guo","year":"2018","journal-title":"ACM Trans. Reconfigurable Technol. Syst."},{"key":"mlstac0ea1bib21","doi-asserted-by":"publisher","first-page":"7823","DOI":"10.1109\/ACCESS.2018.2890150","article-title":"FPGA-based accelerators of deep learning networks for learning and classification: a review","volume":"7","author":"Shawahna","year":"2019","journal-title":"IEEE Access"},{"key":"mlstac0ea1bib22","article-title":"Accelerating CNN inference on FPGAs: a survey","author":"Abdelouahab","year":"2018"},{"key":"mlstac0ea1bib23","doi-asserted-by":"crossref","DOI":"10.1145\/3020078.3021744","article-title":"FINN: A framework for fast, scalable binarized neural network inference","author":"Umuroglu","year":"2017"},{"key":"mlstac0ea1bib24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3242897","article-title":"FINN-R: An end-to-end deep-learning framework for fast exploration of quantized neural networks","volume":"11","author":"Blott","year":"2018","journal-title":"ACM Trans. Reconfigurable Technol. Syst."},{"key":"mlstac0ea1bib25","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4606672","article-title":"Xilinx\/brevitas: Release version 0.4.0","author":"Alessandro","year":"2021"},{"key":"mlstac0ea1bib26","article-title":"fpgaConvNet: a toolflow for mapping diverse convolutional neural networks on embedded FPGAs","author":"Venieris","year":"2017"},{"key":"mlstac0ea1bib27","first-page":"p 291","article-title":"fpgaConvNet: automated mapping of convolutional neural networks on FPGAs","author":"Venieris","year":"2017"},{"key":"mlstac0ea1bib28","first-page":"p 1","article-title":"Latency-driven design for FPGA-based convolutional neural networks","author":"Venieris","year":"2017"},{"key":"mlstac0ea1bib29","first-page":"p 40","article-title":"fpgaConvNet: a framework for mapping convolutional neural networks on FPGAs","author":"Venieris","year":"2016"},{"key":"mlstac0ea1bib30","first-page":"p 675","article-title":"Caffe: convolutional architecture for fast feature embedding","author":"Jia","year":"2014"},{"key":"mlstac0ea1bib31","first-page":"p 152","article-title":"FP-DNN: an automated framework for mapping deep neural networks onto FPGAs with RTL-HLS hybrid templates","author":"Guan","year":"2017"},{"key":"mlstac0ea1bib32","first-page":"p 1","article-title":"From high-level deep neural models to FPGAs","author":"Sharma","year":"2016"},{"key":"mlstac0ea1bib33","first-page":"pp 265","article-title":"Caffeinated FPGAs: FPGA framework for convolutional neural networks","author":"DiCecco","year":"2016"},{"key":"mlstac0ea1bib34","article-title":"Snowflake: a model agnostic accelerator for deep convolutional neural networks","author":"Gokhale","year":"2017"},{"key":"mlstac0ea1bib35","article-title":"Torch7: a Matlab-like environment for machine learning","author":"Collobert","year":"2011"},{"key":"mlstac0ea1bib36","article-title":"A flexible FPGA accelerator for convolutional neural networks","author":"Majumder","year":"2019"},{"key":"mlstac0ea1bib37","doi-asserted-by":"publisher","first-page":"644","DOI":"10.1109\/TNNLS.2018.2852335","article-title":"Nullhop: a flexible convolutional neural network accelerator based on sparse representations of feature maps","volume":"30","author":"Aimar","year":"2019","journal-title":"IEEE Trans. Neural Networks Learn. Syst."},{"key":"mlstac0ea1bib38","first-page":"p 1393","article-title":"Efficient FPGA acceleration of convolutional neural networks using logical-3D compute array","author":"Rahman","year":"2016"},{"key":"mlstac0ea1bib39","author":"","year":"2020"},{"key":"mlstac0ea1bib40","first-page":"pp 19","article-title":"Parallel multi channel convolution using general matrix multiplication","author":"Vasudevan","year":"2017"},{"key":"mlstac0ea1bib41","article-title":"MNIST handwritten digit database","author":"LeCun","year":"2010"},{"key":"mlstac0ea1bib42","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1109\/CVPR.2019.00020","article-title":"Autoaugment: learning augmentation policies from data","author":"Cubuk","year":"2019"},{"key":"mlstac0ea1bib43","article-title":"Improved regularization of convolutional neural networks with cutout","author":"DeVries","year":"2017"},{"key":"mlstac0ea1bib44","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1007\/s42967-020-00085-3","article-title":"Drop-activation: implicit parameter reduction and harmonic regularization","volume":"3","author":"Liang","year":"2021","journal-title":"Commun. Appl. Math. Comput."},{"key":"mlstac0ea1bib45","article-title":"Wide residual networks","author":"Zagoruyko","year":"2017"},{"key":"mlstac0ea1bib46","doi-asserted-by":"publisher","first-page":"1303","DOI":"10.1109\/TCSVT.2017.2654543","article-title":"Residual networks of residual networks: multilevel residual networks","volume":"28","author":"Zhang","year":"2018","journal-title":"IEEE Trans. on Circuits and Systems for Video Technology"},{"key":"mlstac0ea1bib47","article-title":"Convolutional neural networks applied to house numbers digit classification","author":"Sermanet","year":"2012"},{"key":"mlstac0ea1bib48","author":"O\u2019Malley","year":"2019"},{"key":"mlstac0ea1bib49","first-page":"p 448","article-title":"Batch normalization: accelerating deep network training by reducing internal covariate shift","volume":"vol 37","author":"Ioffe","year":"2015"},{"key":"mlstac0ea1bib50","first-page":"p 807","article-title":"Rectified linear units improve restricted Boltzmann machines","author":"Nair","year":"2010"},{"key":"mlstac0ea1bib51","first-page":"p 315","article-title":"Deep sparse rectifier neural networks","volume":"vol 15","author":"Glorot","year":"2011"},{"key":"mlstac0ea1bib52","first-page":"10","volume":"vol 57","author":"Horowitz","year":"2014"},{"key":"mlstac0ea1bib53","author":"Goodfellow","year":"2016"},{"key":"mlstac0ea1bib54","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2015"},{"key":"mlstac0ea1bib55","article-title":"Deep compression: compressing deep neural network with pruning, trained quantization and Huffman coding","author":"Han","year":"2016"},{"key":"mlstac0ea1bib56","first-page":"p 598","author":"LeCun","year":"1990"},{"key":"mlstac0ea1bib57","article-title":"Learning sparse neural networks through l 0 regularization","volume":"vol 12","author":"Louizos","year":"2018"},{"key":"mlstac0ea1bib58","article-title":"Learning both weights and connections for efficient neural networks","author":"Han","year":"2015"},{"key":"mlstac0ea1bib59","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2017.643","article-title":"Designing energy-efficient convolutional neural networks using energy-aware pruning","author":"Yang","year":"2017"},{"key":"mlstac0ea1bib60","article-title":"To prune, or not to prune: exploring the efficacy of pruning for model compression","author":"Zhu","year":"2017"},{"key":"mlstac0ea1bib61","first-page":"6869","article-title":"Quantized neural networks: training neural networks with low precision weights and activations","volume":"18","author":"Hubara","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"mlstac0ea1bib62","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR.2018.00286","article-title":"Quantization and training of neural networks for efficient integer-arithmetic-only inference","author":"Jacob","year":"2018"},{"key":"mlstac0ea1bib63","doi-asserted-by":"publisher","first-page":"p 3123","DOI":"10.5555\/2969442.2969588","author":"Courbariaux","year":"2015"}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,13]],"date-time":"2021-12-13T14:57:22Z","timestamp":1639407442000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/ac0ea1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,16]]},"references-count":63,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,7,16]]},"published-print":{"date-parts":[[2021,12,1]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/ac0ea1","relation":{},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,16]]},"assertion":[{"value":"Fast convolutional neural networks on FPGAs with hls4ml","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2021 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2021-01-15","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-06-25","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2021-07-16","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}