{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:25:25Z","timestamp":1750220725949,"version":"3.41.0"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,1,30]],"date-time":"2020-01-30T00:00:00Z","timestamp":1580342400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000015","name":"US Department of Energy","doi-asserted-by":"crossref","award":["SC0017030"],"award-info":[{"award-number":["SC0017030"]}],"id":[{"id":"10.13039\/100000015","id-type":"DOI","asserted-by":"crossref"}]},{"name":"NRC Associate Fellowship Award"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Emerg. Technol. Comput. Syst."],"published-print":{"date-parts":[[2020,4,30]]},"abstract":"<jats:p>\n            Deep neural network (DNN) models are being expanded to a broader range of applications. The computational capability of traditional hardware platforms cannot accommodate the growth of model complexity. Among recent technologies to accelerate DNN, resistive memory (ReRAM)-based processing-in-memory (PIM) emerged as a promising solution for DNN inference due to its high efficiency for matrix-based computation. We face two major technical challenges in extending the use of ReRAM-based accelerators for training: (1) full-precision data is essential in back-propagation; (2) the need to support both feed-forward and back-propagation aggravates the data-movement burden. We propose a heterogeneous architecture named as 3D-ReG, which leverages full-precision GPU to ensure training accuracy and low-overhead 3D integration to provide low-cost data movements. Moreover, we introduce\n            <jats:italic>conservative<\/jats:italic>\n            and\n            <jats:italic>aggressive<\/jats:italic>\n            task-mapping schemes, which partition the computation phases in different ways to balance execution efficiency and training accuracy. We evaluate 3D-ReG implemented with two 3D integration technologies, through-silicon vias (TSVs) and monolithic inter-tier vias (MIVs), and compare them with GPU-only and PIM-only counterparts. Various GPU-only platforms using two main-memory technologies (DRAM, ReRAM) and three interconnect technologies (2D, TSV, MIV) are evaluated as well. Experimental results show that 3D-ReG can achieve on average 5.64\u00d7 training speedup and 3.56\u00d7 higher energy efficiency compared with the GPU with DRAM as main memory, at the cost of 0.05%\u20133.39% accuracy drop. We define a new metric, gain-loss ratio (GLR), which quantitatively evaluates the capability of a DNN training hardware in terms of the model accuracy and hardware efficiency. The results of our comparison show that the\n            <jats:italic>aggressive<\/jats:italic>\n            task-mapping scheme on MIV-based 3D-ReG outperforms the other methods.\n          <\/jats:p>","DOI":"10.1145\/3375699","type":"journal-article","created":{"date-parts":[[2020,3,2]],"date-time":"2020-03-02T22:50:06Z","timestamp":1583189406000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["3D-ReG"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0732-2267","authenticated-orcid":false,"given":"Bing","family":"Li","sequence":"first","affiliation":[{"name":"Capital Normal University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Janardhan Rao","family":"Doppa","sequence":"additional","affiliation":[{"name":"Washington State University, Pullman, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Partha Pratim","family":"Pande","sequence":"additional","affiliation":[{"name":"Washington State University, Pullman, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Krishnendu","family":"Chakrabarty","sequence":"additional","affiliation":[{"name":"Duke University, Durham, NC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joe X.","family":"Qiu","sequence":"additional","affiliation":[{"name":"Duke University and Army Research Office, Research Triangle Park, Durham, NC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hai (Helen)","family":"Li","sequence":"additional","affiliation":[{"name":"Duke University, Durham, NC"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,1,30]]},"reference":[{"volume-title":"Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software. 163--174","year":"2009","author":"Ali","key":"e_1_2_1_1_1"},{"key":"e_1_2_1_2_1","unstructured":"Tor M. Aamodt Wilson W. L. Fung Inderpreet Singh Ahmed El-Shafiey Jimmy Kwa Tayler Hetherington Ayub Gubran Andrew Boktor Tim Rogers Ali Bakhoda etal 2012. GPGPU-Sim Manual. Retrieved from http:\/\/gpgpu-sim.org\/manual\/.  Tor M. Aamodt Wilson W. L. Fung Inderpreet Singh Ahmed El-Shafiey Jimmy Kwa Tayler Hetherington Ayub Gubran Andrew Boktor Tim Rogers Ali Bakhoda et al. 2012. GPGPU-Sim Manual. Retrieved from http:\/\/gpgpu-sim.org\/manual\/."},{"volume-title":"Proceedings of the ACM\/IEEE 42nd International Symposium on Computer Architecture (ISCA\u201915)","author":"Akin Berkin","key":"e_1_2_1_3_1"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCD.2009.5413147"},{"key":"e_1_2_1_5_1","unstructured":"AMD. 2017. Radeon\u2019s next-generation Vega architecture. Retrieved from https:\/\/radeon.com\/_downloads\/vega-whitepaper-11.6.17.pdf.  AMD. 2017. Radeon\u2019s next-generation Vega architecture. Retrieved from https:\/\/radeon.com\/_downloads\/vega-whitepaper-11.6.17.pdf."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3195970.3196009"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2017.2752706"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2016.7446049"},{"key":"e_1_2_1_9_1","first-page":"89","article-title":"Neuromorphic computing using non-volatile memory","author":"Burr Geoffrey W.","year":"2017","journal-title":"Adv. Phys."},{"volume-title":"Proceedings of the 5th IEEE International Memory Workshop. 116--119","year":"2013","author":"Cabout T.","key":"e_1_2_1_10_1"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3195970.3196071"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISLPED.2017.8009175"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASPDAC.2018.8297302"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2541940.2541967"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001177"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/IEDM.2015.7409722"},{"volume-title":"Proceedings of the International Electron Devices Meeting. IEEE, 19--4.","author":"Cheng Chun-Hu","key":"e_1_2_1_17_1"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3061639.3062326"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001140"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295275"},{"key":"e_1_2_1_21_1","unstructured":"Hybrid Memory Cube Consortium. 2015. Hybrid memory cube specification 2.1. Retrieved from http:\/\/hybridmemorycube.org\/files\/SiteDownloads\/HMC30GVSR_HMCC_Specification_Rev2.1_20151105.pdf.  Hybrid Memory Cube Consortium. 2015. Hybrid memory cube specification 2.1. Retrieved from http:\/\/hybridmemorycube.org\/files\/SiteDownloads\/HMC30GVSR_HMCC_Specification_Rev2.1_20151105.pdf."},{"volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems. 3123--3131","year":"2015","author":"Courbariaux Matthieu","key":"e_1_2_1_22_1"},{"key":"e_1_2_1_23_1","unstructured":"Matthieu Courbariaux Itay Hubara etal 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to + 1 or &minus;1. arXiv preprint arXiv:1602.02830 (2016).  Matthieu Courbariaux Itay Hubara et al. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to + 1 or &minus;1. arXiv preprint arXiv:1602.02830 (2016)."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/514191.514197"},{"volume-title":"Proceedings of the 2nd Workshop on Near-Data Processing (WoNDP\u201914)","author":"Eckert Yasuko","key":"e_1_2_1_25_1"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2324876.2324879"},{"volume-title":"Caffe style. Retrieved on","year":"2018","author":"Donahue Jeff","key":"e_1_2_1_27_1"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSCC.2014.6757460"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3037697.3037702"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2016.00333"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915)","volume":"37","author":"Gupta Suyog","year":"2015"},{"volume-title":"Hardware-oriented approximation of convolutional neural networks. arXiv preprint arXiv:1604.03168","year":"2016","author":"Gysel Philipp","key":"e_1_2_1_32_1"},{"volume-title":"Proceedings of the 43rd International Symposium on Computer Architecture (ISCA\u201916)","year":"2016","author":"Han Song","key":"e_1_2_1_33_1"},{"volume-title":"Dally","year":"2015","author":"Han Song","key":"e_1_2_1_34_1"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"volume-title":"Proceedings of the Symposium on VLSI Technology. IEEE, T166--T167","year":"2013","author":"Hsu Chung-Wei","key":"e_1_2_1_36_1"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2296777"},{"volume-title":"Proceedings of the 53rd Design Automation Conference (DAC\u201916)","author":"Hu Miao","key":"e_1_2_1_38_1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2018.011441561"},{"key":"e_1_2_1_40_1","unstructured":"JEDEC Standard. 2013. High Bandwidth Memory (HBM) DRAM JESD 235. https:\/\/www.jedec.org\/standards-documents\/docs\/jesd235a.  JEDEC Standard. 2013. High Bandwidth Memory (HBM) DRAM JESD 235. https:\/\/www.jedec.org\/standards-documents\/docs\/jesd235a."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3079856.3080246"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001178"},{"volume-title":"NeuroTrainer: An intelligent memory module for deep learning training. arXiv preprint arXiv:1710.04347","year":"2017","author":"Kim Duckhwan","key":"e_1_2_1_44_1"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2018.04.007"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISLPED.2017.8009157"},{"key":"e_1_2_1_47_1","first-page":"9","article-title":"Scalable processors in the billion-transistor era","volume":"30","author":"Kozyrakis Christoforos E.","year":"1997","journal-title":"IRAM. Computer"},{"volume-title":"Hinton","year":"2012","author":"Krizhevsky Alex","key":"e_1_2_1_49_1"},{"key":"e_1_2_1_50_1","unstructured":"Yann LeCun. 1998. The MNIST database of handwritten digits. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist\/(1998).  Yann LeCun. 1998. The MNIST database of handwritten digits. Retrieved from http:\/\/yann.lecun.com\/exdb\/mnist\/(1998)."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2485922.2485964"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISLPED.2013.6629302"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASPDAC.2018.8297378"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISQED.2012.6187545"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/MICRO.2018.00059"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2744769.2744900"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA.2008.15"},{"volume-title":"Inside Volta: The World\u2019s Most Advanced Data Center GPU.","year":"2017","author":"Harris Mark","key":"e_1_2_1_58_1"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2015.2409732"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAD.2013.6691092"},{"volume-title":"n.d. Nvidia GeForce GTX480 Specifications. Retrieved on","year":"2018","key":"e_1_2_1_61_1"},{"key":"e_1_2_1_62_1","unstructured":"Nvidia. 2017. The power of GeForce GTX 1080. Retrieved from https:\/\/www.nvidia.com\/en-us\/geforce\/products\/10series\/geforce-gtx-1080\/.  Nvidia. 2017. The power of GeForce GTX 1080. Retrieved from https:\/\/www.nvidia.com\/en-us\/geforce\/products\/10series\/geforce-gtx-1080\/."},{"volume-title":"Proceedings of the 50th IEEE\/ACM International Symposium on Microarchitecture (MICRO\u201917)","author":"O\u2019Connor Mike","key":"e_1_2_1_63_1"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593069.2593188"},{"volume-title":"Quantizing convolutional neural networks for low-power high-throughput inference engines. arXiv preprint arXiv:1805.07941","year":"2018","author":"Settle Sean O.","key":"e_1_2_1_65_1"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3007787.3001139"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463209.2488866"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-bioeng-071516-044442"},{"volume-title":"Three-dimensional integration of nanotechnologies for computing and data storage on a single chip. Nature 547, 7661","year":"2017","author":"Shulaker Max M.","key":"e_1_2_1_69_1"},{"volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556","year":"2014","author":"Simonyan Karen","key":"e_1_2_1_70_1"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2017.55"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSSC.2010.2074070"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3140659.3080244"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3195970.3196116"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.5555\/3157096.3157329"},{"volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS\u201917)","year":"2017","author":"Wen Wei","key":"e_1_2_1_76_1"},{"volume-title":"Proceedings of the 16th International Symposium on High-Performance Computer Architecture. 1--12","year":"2010","author":"Woo Dong Hyuk","key":"e_1_2_1_77_1"},{"volume-title":"Proceedings of the IEEE International Electron Devices Meeting (IEDM\u201915)","year":"2015","author":"Wu Tsung-Ta","key":"e_1_2_1_78_1"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNANO.2017.2731871"},{"volume-title":"Yuxin Wu et al","year":"2016","author":"Zhou Shuchang","key":"e_1_2_1_80_1"}],"container-title":["ACM Journal on Emerging Technologies in Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375699","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3375699","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:15Z","timestamp":1750199895000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375699"}},"subtitle":["A 3D ReRAM-based Heterogeneous Architecture for Training Deep Neural Networks"],"short-title":[],"issued":{"date-parts":[[2020,1,30]]},"references-count":79,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,4,30]]}},"alternative-id":["10.1145\/3375699"],"URL":"https:\/\/doi.org\/10.1145\/3375699","relation":{},"ISSN":["1550-4832","1550-4840"],"issn-type":[{"type":"print","value":"1550-4832"},{"type":"electronic","value":"1550-4840"}],"subject":[],"published":{"date-parts":[[2020,1,30]]},"assertion":[{"value":"2019-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}