{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T11:03:14Z","timestamp":1770894194457,"version":"3.50.1"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,7,30]],"date-time":"2018-07-30T00:00:00Z","timestamp":1532908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"crossref"}]},{"name":"ADEPT Lab industrial sponsors"},{"name":"Intel Science and Technology Center for Agile Design"},{"name":"NSF\/Intel Partnership on Computer Assisted Programming for Heterogeneous Architectures","award":["CCF-1723445"],"award-info":[{"award-number":["CCF-1723445"]}]},{"DOI":"10.13039\/501100004405","name":"Toyota","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100004405","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Siemens"},{"DOI":"10.13039\/100018058","name":"SK Hynix","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100018058","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2018,8,31]]},"abstract":"<jats:p>\n            Gradient-based optimization has enabled dramatic advances in computational imaging through techniques like deep learning and nonlinear optimization. These methods require gradients not just of simple mathematical functions, but of general\n            <jats:italic>programs<\/jats:italic>\n            which encode complex transformations of images and graphical data. Unfortunately, practitioners have traditionally been limited to either hand-deriving gradients of complex computations, or composing programs from a limited set of coarse-grained operators in deep learning frameworks. At the same time, writing programs with the level of performance needed for imaging and deep learning is prohibitively difficult for most programmers.\n          <\/jats:p>\n          <jats:p>We extend the image processing language Halide with general reverse-mode automatic differentiation (AD), and the ability to automatically optimize the implementation of gradient computations. This enables automatic computation of the gradients of arbitrary Halide programs, at high performance, with little programmer effort. A key challenge is to structure the gradient code to retain parallelism. We define a simple algorithm to automatically schedule these pipelines, and show how Halide's existing scheduling primitives can express and extend the key AD optimization of \"checkpointing.\"<\/jats:p>\n          <jats:p>Using this new tool, we show how to easily define new neural network layers which automatically compile to high-performance GPU implementations, and how to solve nonlinear inverse problems from computational imaging. Finally, we show how differentiable programming enables dramatically improving the quality of even traditional, feed-forward image processing algorithms, blurring the distinction between classical and deep methods.<\/jats:p>","DOI":"10.1145\/3197517.3201383","type":"journal-article","created":{"date-parts":[[2018,7,31]],"date-time":"2018-07-31T15:56:23Z","timestamp":1533052583000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":63,"title":["Differentiable programming for image processing and deep learning in halide"],"prefix":"10.1145","volume":"37","author":[{"given":"Tzu-Mao","family":"Li","sequence":"first","affiliation":[{"name":"MIT CSAIL"}]},{"given":"Micha\u00ebl","family":"Gharbi","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}]},{"given":"Andrew","family":"Adams","sequence":"additional","affiliation":[{"name":"Facebook AI Research"}]},{"given":"Fr\u00e9do","family":"Durand","sequence":"additional","affiliation":[{"name":"MIT CSAIL"}]},{"given":"Jonathan","family":"Ragan-Kelley","sequence":"additional","affiliation":[{"name":"UC Berkeley &amp; Google"}]}],"member":"320","published-online":{"date-parts":[[2018,7,30]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  Mart\u00edn Abadi Ashish Agarwal Paul Barham Eugene Brevdo Zhifeng Chen Craig Citro Greg S. Corrado Andy Davis Jeffrey Dean Matthieu Devin Sanjay Ghemawat Ian Goodfellow Andrew Harp Geoffrey Irving Michael Isard Yangqing Jia Rafal Jozefowicz Lukasz Kaiser Manjunath Kudlur Josh Levenberg Dan Man\u00e9 Rajat Monga Sherry Moore Derek Murray Chris Olah Mike Schuster Jonathon Shlens Benoit Steiner Ilya Sutskever Kunal Talwar Paul Tucker Vincent Vanhoucke Vijay Vasudevan Fernanda Vi\u00e9gas Oriol Vinyals Pete Warden Martin Wattenberg Martin Wicke Yuan Yu and Xiaoqiang Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46487-9_38"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-92bf1922-003"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/143242.143335"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995332"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276506"},{"key":"e_1_2_2_7_1","volume-title":"cuDNN: Efficient Primitives for Deep Learning. arXiv preprint arXiv:1410.0759","author":"Chetlur Sharan","year":"2014","unstructured":"Sharan Chetlur , Cliff Woolley , Philippe Vandermersch , Jonathan Cohen , John Tran , Bryan Catanzaro , and Evan Shelhamer . 2014. cuDNN: Efficient Primitives for Deep Learning. arXiv preprint arXiv:1410.0759 ( 2014 ). Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. cuDNN: Efficient Primitives for Deep Learning. arXiv preprint arXiv:1410.0759 (2014)."},{"key":"e_1_2_2_8_1","doi-asserted-by":"crossref","unstructured":"J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.  J. Deng W. Dong R. Socher L.-J. Li K. Li and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132188"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/358669.358692"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00371-014-0966-x"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.265"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982399"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073592"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2010.00765.x"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/229473.229474"},{"key":"e_1_2_2_17_1","unstructured":"Andreas Griewank and Shawn Reese. 1991. On the Calculation of Jacobian Matrices by the Markowitz Rule. In Automatic Differentiation of Algorithms: Theory Implementation and Application Andreas Griewank and George F. Corliss (Eds.). 126--135.  Andreas Griewank and Shawn Reese. 1991. On the Calculation of Jacobian Matrices by the Markowitz Rule. In Automatic Differentiation of Algorithms: Theory Implementation and Application Andreas Griewank and George F. Corliss (Eds.). 126--135."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/1455489"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1276377.1276512"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2450153.2450158"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925875"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661229.2661260"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2004.838691"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2560359"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(81)90024-2"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925974"},{"key":"e_1_2_2_27_1","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR).","author":"Ilg E.","unstructured":"E. Ilg , N. Mayer , T. Saikia , M. Keuper , A. Dosovitskiy , and T. Brox . 2017. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. 2017. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR)."},{"key":"e_1_2_2_28_1","unstructured":"Max Jaderberg Karen Simonyan Andrew Zisserman etal 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025.   Max Jaderberg Karen Simonyan Andrew Zisserman et al. 2015. Spatial transformer networks. In Advances in Neural Information Processing Systems. 2017--2025."},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_2_2_30_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma Diederik","year":"2014","unstructured":"Diederik Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)."},{"key":"e_1_2_2_31_1","volume-title":"DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064","author":"Kupyn Orest","year":"2017","unstructured":"Orest Kupyn , Volodymyr Budzan , Mykola Mykhailych , Dmytro Mishkin , and Jiri Matas . 2017. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064 ( 2017 ). Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2017. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064 (2017)."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/647406.724265"},{"key":"e_1_2_2_33_1","unstructured":"Gunther Lange. 1957. Gauss type photographic objective containing two outer collective and two inner dispersive members. U.S. Patent 2 799 207 A.  Gunther Lange. 1957. Gauss type photographic objective containing two outer collective and two inner dispersive members. U.S. Patent 2 799 207 A."},{"key":"e_1_2_2_34_1","unstructured":"Seppo Linnainmaa. 1970. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's thesis. Univ. Helsinki.  Seppo Linnainmaa. 1970. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master's thesis. Univ. Helsinki."},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:VISI.0000029664.99615.94"},{"key":"e_1_2_2_36_1","volume-title":"Deep Photo Style Transfer. arXiv preprint arXiv:1703.07511","author":"Luan Fujun","year":"2017","unstructured":"Fujun Luan , Sylvain Paris , Eli Shechtman , and Kavita Bala . 2017. Deep Photo Style Transfer. arXiv preprint arXiv:1703.07511 ( 2017 ). Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. 2017. Deep Photo Style Transfer. arXiv preprint arXiv:1703.07511 (2017)."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897824.2925952"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2786763.2694364"},{"key":"e_1_2_2_39_1","unstructured":"Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).  Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2185520.2185528"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462176"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2005.160"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167-2789(92)90242-F"},{"key":"e_1_2_2_44_1","volume-title":"Parallel Distributed Processing: Explorations in the Microstructure of Cognition","volume":"1","author":"Rumelhart D. E.","unstructured":"D. E. Rumelhart , G. E. Hinton , and R. J. Williams . 1986 . Parallel Distributed Processing: Explorations in the Microstructure of Cognition , Vol. 1 . Chapter Learning Internal Representations by Error Propagation, 318--362. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1. Chapter Learning Internal Representations by Error Propagation, 318--362."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.5555\/3049832.3049863"},{"key":"e_1_2_2_46_1","volume-title":"Deep Image Prior. arXiv preprint arXiv:1711.10925","author":"Ulyanov Dmitry","year":"2017","unstructured":"Dmitry Ulyanov , Andrea Vedaldi , and Victor Lempitsky . 2017. Deep Image Prior. arXiv preprint arXiv:1711.10925 ( 2017 ). Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2017. Deep Image Prior. arXiv preprint arXiv:1711.10925 (2017)."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/0898-1221(85)90188-9"},{"key":"e_1_2_2_48_1","volume-title":"System modeling and optimization","author":"Werbos Paul J","unstructured":"Paul J Werbos . 1982. Applications of advances in nonlinear sensitivity analysis . In System modeling and optimization . Springer , 762--770. Paul J Werbos. 1982. Applications of advances in nonlinear sensitivity analysis. In System modeling and optimization. Springer, 762--770."},{"key":"e_1_2_2_49_1","unstructured":"Alexander B Wiltschko Bart van Merri\u00c3\u0144nboer and Dan Moldovan. 2017. Tangent: automatic differentiation using source code transformation in Python.  Alexander B Wiltschko Bart van Merri\u00c3\u0144nboer and Dan Moldovan. 2017. Tangent: automatic differentiation using source code transformation in Python."},{"key":"e_1_2_2_50_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning (ICML-15)","author":"Xu Li","year":"2015","unstructured":"Li Xu , Jimmy Ren , Qiong Yan , Renjie Liao , and Jiaya Jia . 2015 . Deep edge-aware filters . In Proceedings of the 32nd International Conference on Machine Learning (ICML-15) . 1669--1678. Li Xu, Jimmy Ren, Qiong Yan, Renjie Liao, and Jiaya Jia. 2015. Deep edge-aware filters. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15). 1669--1678."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980179.2982403"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2662206"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46487-9_40"},{"key":"e_1_2_2_55_1","volume-title":"Image registration methods: a survey. Image and vision computing 21, 11","author":"Zitova Barbara","year":"2003","unstructured":"Barbara Zitova and Jan Flusser . 2003. Image registration methods: a survey. Image and vision computing 21, 11 ( 2003 ), 977--1000. Barbara Zitova and Jan Flusser. 2003. Image registration methods: a survey. Image and vision computing 21, 11 (2003), 977--1000."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3197517.3201383","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3197517.3201383","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:06:59Z","timestamp":1750212419000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3197517.3201383"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,30]]},"references-count":54,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,8,31]]}},"alternative-id":["10.1145\/3197517.3201383"],"URL":"https:\/\/doi.org\/10.1145\/3197517.3201383","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,7,30]]},"assertion":[{"value":"2018-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}