{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T12:37:29Z","timestamp":1794573449363,"version":"4.9.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"ICFP","license":[{"start":{"date-parts":[[2018,7,30]],"date-time":"2018-07-30T00:00:00Z","timestamp":1532908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2018,7,30]]},"abstract":"<jats:p>Automatic differentiation (AD) in reverse mode (RAD) is a central component of deep learning and other uses of large-scale optimization. Commonly used RAD algorithms such as backpropagation, however, are complex and stateful, hindering deep understanding, improvement, and parallel execution. This paper develops a simple, generalized AD algorithm calculated from a simple, natural specification. The general algorithm is then specialized by varying the representation of derivatives. In particular, applying well-known constructions to a naive representation yields two RAD algorithms that are far simpler than previously known. In contrast to commonly used RAD implementations, the algorithms defined here involve no graphs, tapes, variables, partial derivatives, or mutation. They are inherently parallel-friendly, correct by construction, and usable directly from an existing programming language with no need for new data types or programming style, thanks to use of an AD-agnostic compiler plugin.<\/jats:p>","DOI":"10.1145\/3236765","type":"journal-article","created":{"date-parts":[[2018,7,31]],"date-time":"2018-07-31T19:41:18Z","timestamp":1533066078000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":62,"title":["The simple essence of automatic differentiation"],"prefix":"10.1145","volume":"2","author":[{"given":"Conal","family":"Elliott","sequence":"first","affiliation":[{"name":"Target, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,7,30]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)","author":"Abadi Martin","year":"2016","unstructured":"Martin Abadi , Paul Barham , Jianmin Chen , Zhifeng Chen , Andy Davis , Jeffrey Dean , Matthieu Devin , Sanjay Ghemawat , Geoffrey Irving , Michael Isard , Manjunath Kudlur , Josh Levenberg , Rajat Monga , Sherry Moore , Derek G. Murray , Benoit Steiner , Paul Tucker , Vijay Vasudevan , Pete Warden , Martin Wicke , Yuan Yu , and Xiaoqiang Zheng . 2016 . TensorFlow: A system for large-scale machine learning . In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) . 265\u2013283. Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning . In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265\u2013283."},{"key":"e_1_2_2_2_1","volume-title":"Compiling with Continuations","author":"Appel Andrew W.","unstructured":"Andrew W. Appel . 2007. Compiling with Continuations . Cambridge University Press . Andrew W. Appel. 2007. Compiling with Continuations. Cambridge University Press."},{"key":"e_1_2_2_3_1","volume-title":"Category theory. Oxford Logic Guides","author":"Awodey Steve","unstructured":"Steve Awodey . 2006. Category theory. Oxford Logic Guides , Vol. 49 . Oxford University Press . Steve Awodey. 2006. Category theory. Oxford Logic Guides, Vol. 49. Oxford University Press."},{"key":"e_1_2_2_4_1","volume-title":"The Algebra of Programming","author":"Bird Richard","unstructured":"Richard Bird and Oege de Moor . 1996. The Algebra of Programming . Prentice-Hall . Richard Bird and Oege de Moor. 1996. The Algebra of Programming . Prentice-Hall."},{"key":"e_1_2_2_5_1","unstructured":"Max Bolingbroke. 2011. Constraint kinds for GHC. Blog post. http:\/\/blog.omega- prime.co.uk\/2011\/09\/10\/ constraint- kinds- for- ghc\/ .  Max Bolingbroke. 2011. Constraint kinds for GHC. Blog post. http:\/\/blog.omega- prime.co.uk\/2011\/09\/10\/ constraint- kinds- for- ghc\/ ."},{"key":"e_1_2_2_6_1","unstructured":"Fran\u00e7ois Chollet. 2016. Keras resources. GitHub repository. https:\/\/github.com\/fchollet  Fran\u00e7ois Chollet. 2016. Keras resources. GitHub repository. https:\/\/github.com\/fchollet"},{"key":"e_1_2_2_7_1","volume-title":"Introduction to Algorithms","author":"Cormen Thomas H.","unstructured":"Thomas H. Cormen , Charles E. Leiserson , Ronald L. Rivest , and Clifford Stein . 2001. Introduction to Algorithms , Third Edition. The MIT Press and McGraw-Hill Book Company . Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2001. Introduction to Algorithms, Third Edition. The MIT Press and McGraw-Hill Book Company."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/773184.773202"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1596550.1596579"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3110271"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236765"},{"key":"e_1_2_2_12_1","volume-title":"Backprop as functor: A compositional perspective on supervised learning . CoRR abs\/1711.10455","author":"Fong Brendan","year":"2017","unstructured":"Brendan Fong , David I. Spivak , and R\u00e9my Tuy\u00e9ras . 2017. Backprop as functor: A compositional perspective on supervised learning . CoRR abs\/1711.10455 ( 2017 ). Brendan Fong, David I. Spivak, and R\u00e9my Tuy\u00e9ras. 2017. Backprop as functor: A compositional perspective on supervised learning . CoRR abs\/1711.10455 (2017)."},{"key":"e_1_2_2_13_1","series-title":"Lecture Notes in Computer Science","volume-title":"Algebraic and Coalgebraic Methods in the Mathematics of Program Construction","author":"Gibbons Jeremy","unstructured":"Jeremy Gibbons . 2002. Calculating functional programs . In Algebraic and Coalgebraic Methods in the Mathematics of Program Construction . Lecture Notes in Computer Science , Vol. 2297 . Springer-Verlag . Jeremy Gibbons. 2002. Calculating functional programs . In Algebraic and Coalgebraic Methods in the Mathematics of Program Construction. Lecture Notes in Computer Science, Vol. 2297. Springer-Verlag."},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1596638.1596653"},{"key":"e_1_2_2_15_1","volume-title":"Deep Learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep Learning . MIT Press . Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press."},{"key":"e_1_2_2_16_1","unstructured":"Andreas Griewank. 1989. On Automatic Differentiation. In In Mathematical Programming: Recent Developments and Applications.  Andreas Griewank. 1989. On Automatic Differentiation. In In Mathematical Programming: Recent Developments and Applications."},{"key":"e_1_2_2_17_1","volume-title":"Principles and Techniques of Algorithmic Differentiation","author":"Griewank Andreas","unstructured":"Andreas Griewank and Andrea Walther . 2008. Evaluating Derivatives . Principles and Techniques of Algorithmic Differentiation ( second ed.). Society for Industrial and Applied Mathematics . Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives. Principles and Techniques of Algorithmic Differentiation (second ed.). Society for Industrial and Applied Mathematics."},{"key":"e_1_2_2_18_1","unstructured":"Joe Hermaszewski and Ben Gamari. 2017. vector-sized. http:\/\/github.com\/expipiplus1\/vector- sized Haskell library.  Joe Hermaszewski and Ben Gamari. 2017. vector-sized. http:\/\/github.com\/expipiplus1\/vector- sized Haskell library."},{"key":"e_1_2_2_19_1","volume-title":"2nd Workshop on Generic Programming. 17\u201332","author":"Hinze Ralf","year":"2000","unstructured":"Ralf Hinze . 2000 . Memo functions, polytypically! . In 2nd Workshop on Generic Programming. 17\u201332 . Ralf Hinze. 2000. Memo functions, polytypically! . In 2nd Workshop on Generic Programming. 17\u201332."},{"key":"e_1_2_2_20_1","unstructured":"T. C. Hu and M. T. Shing. 1981. Computation of matrix chain products Part I Part II. Technical Report STAN-CS-TR-81-875. Stanford University Department of Computer Science.   T. C. Hu and M. T. Shing. 1981. Computation of matrix chain products Part I Part II. Technical Report STAN-CS-TR-81-875. Stanford University Department of Computer Science."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2647868.2654889"},{"key":"e_1_2_2_22_1","volume-title":"Scottish Workshop on Functional Programming.","author":"Karczmarczuk Jerzy","year":"1999","unstructured":"Jerzy Karczmarczuk . 1999 . Functional coding of differential forms . In Scottish Workshop on Functional Programming. Jerzy Karczmarczuk. 1999. Functional coding of differential forms . In Scottish Workshop on Functional Programming."},{"key":"e_1_2_2_23_1","unstructured":"Jerzy Karczmarczuk. 2000. Adjoint codes in functional framework .  Jerzy Karczmarczuk. 2000. Adjoint codes in functional framework ."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1011501232197"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1291151.1291179"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1017472.1017481"},{"key":"e_1_2_2_27_1","unstructured":"Edward Kmett. 2011. The adjunctions package. https:\/\/hackage.haskell.org\/package\/adjunctions . Haskell library.  Edward Kmett. 2011. The adjunctions package. https:\/\/hackage.haskell.org\/package\/adjunctions . Haskell library."},{"key":"e_1_2_2_28_1","unstructured":"Edward Kmett Barak Pearlmutter and Jeffrey Mark Siskind. 2010. The ad package. https:\/\/hackage.haskell.org\/package\/ad . Haskell library.  Edward Kmett Barak Pearlmutter and Jeffrey Mark Siskind. 2010. The ad package. https:\/\/hackage.haskell.org\/package\/ad . Haskell library."},{"key":"e_1_2_2_29_1","volume-title":"To H.B. Curry: Essays on Combinatory Logic, Lambda Calculus, and Formalism","author":"Lambek Joachim","unstructured":"Joachim Lambek . 1980. From \u03bb-calculus to cartesian closed categories . In To H.B. Curry: Essays on Combinatory Logic, Lambda Calculus, and Formalism , J.P. Seldin and J.R. Hindley (Eds.). Academic Press . Joachim Lambek. 1980. From \u03bb-calculus to cartesian closed categories. In To H.B. Curry: Essays on Combinatory Logic, Lambda Calculus, and Formalism, J.P. Seldin and J.R. Hindley (Eds.). Academic Press."},{"key":"e_1_2_2_30_1","doi-asserted-by":"crossref","unstructured":"Joachim Lambek. 1986. Cartesian closed categories and typed lambda-calculi. In Thirteenth Spring School of the LITP on Combinators and Functional Programming Languages. 136\u2013175.   Joachim Lambek. 1986. Cartesian closed categories and typed lambda-calculi. In Thirteenth Spring School of the LITP on Combinators and Functional Programming Languages. 136\u2013175.","DOI":"10.1007\/3-540-17184-3_44"},{"key":"e_1_2_2_31_1","volume-title":"Linear Algebra","author":"Lang Serge","unstructured":"Serge Lang . 1987. Linear Algebra ( 3 rd ed.). Springer-Verlag . Serge Lang. 1987. Linear Algebra (3rd ed.). Springer-Verlag.","edition":"3"},{"key":"e_1_2_2_32_1","volume-title":"Schanuel","author":"William Lawvere F.","year":"2009","unstructured":"F. William Lawvere and Stephen H . Schanuel . 2009 . Conceptual Mathematics : A First Introduction to Categories (2nd ed.). Cambridge University Press . F. William Lawvere and Stephen H. Schanuel. 2009. Conceptual Mathematics: A First Introduction to Categories (2nd ed.). Cambridge University Press."},{"key":"e_1_2_2_33_1","volume-title":"Deep learning. Nature 521, 7553 (27 5","author":"Lecun Yann","year":"2015","unstructured":"Yann Lecun , Yoshua Bengio , and Geoffrey Hinton . 2015. Deep learning. Nature 521, 7553 (27 5 2015 ), 436\u2013444. Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (27 5 2015), 436\u2013444."},{"key":"e_1_2_2_34_1","volume-title":"The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master\u2019s thesis","author":"Linnainmaa Seppo","unstructured":"Seppo Linnainmaa . 1970. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master\u2019s thesis . University of Helsinki . Seppo Linnainmaa. 1970. The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master\u2019s thesis. University of Helsinki."},{"key":"e_1_2_2_35_1","volume-title":"Categories for the Working Mathematician","author":"Lane Saunders Mac","unstructured":"Saunders Mac Lane . 1998. Categories for the Working Mathematician . Springer New York . Saunders Mac Lane. 1998. Categories for the Working Mathematician. Springer New York."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2012.07.012"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1863523.1863529"},{"key":"e_1_2_2_38_1","unstructured":"Jos\u00e9 Pedro Magalh\u00e3es etal 2011. GHC.Generics. https:\/\/wiki.haskell.org\/GHC.Generics Haskell wiki.  Jos\u00e9 Pedro Magalh\u00e3es et al. 2011. GHC.Generics. https:\/\/wiki.haskell.org\/GHC.Generics Haskell wiki."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/3114201.3114717"},{"key":"e_1_2_2_40_1","unstructured":"Chris Olah. 2015. Neural networks types and functional programming. Blog post. http:\/\/colah.github.io\/posts\/ 2015- 09- NN- Types- FP\/ .  Chris Olah. 2015. Neural networks types and functional programming. Blog post. http:\/\/colah.github.io\/posts\/ 2015- 09- NN- Types- FP\/ ."},{"key":"e_1_2_2_41_1","unstructured":"Jos\u00e9 Nuno Oliveira. 2018. Program Design by Calculation . Draft of textbook in preparation.  Jos\u00e9 Nuno Oliveira. 2018. Program Design by Calculation . Draft of textbook in preparation."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1190216.1190242"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1330017.1330018"},{"key":"e_1_2_2_44_1","doi-asserted-by":"crossref","unstructured":"Simon L. Peyton Jones Simon Marlow and Conal Elliott. 1999. Stretching the storage manager: Weak pointers and stable names in Haskell . In Implementation of Functional Languages.   Simon L. Peyton Jones Simon Marlow and Conal Elliott. 1999. Stretching the storage manager: Weak pointers and stable names in Haskell . In Implementation of Functional Languages.","DOI":"10.1007\/10722298_3"},{"key":"e_1_2_2_45_1","first-page":"123","article-title":"Differentiability properties of computable functions\u2014A summary","volume":"4","author":"Pour-El Marian Boykan","year":"1978","unstructured":"Marian Boykan Pour-El and Ian Richards . 1978 . Differentiability properties of computable functions\u2014A summary . Acta Cybernetica 4 , 1 (1978), 123 \u2013 125 . Marian Boykan Pour-El and Ian Richards. 1978. Differentiability properties of computable functions\u2014A summary . Acta Cybernetica 4, 1 (1978), 123\u2013125.","journal-title":"Acta Cybernetica"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9947-1983-0682717-1"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-10861-0"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/800194.805852"},{"key":"e_1_2_2_49_1","volume-title":"Category Theory in Context","author":"Riehl Emily","unstructured":"Emily Riehl . 2016. Category Theory in Context . Dover Publications . Emily Riehl. 2016. Category Theory in Context. Dover Publications."},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.2140\/pjm.1966.17.497"},{"key":"e_1_2_2_51_1","volume-title":"Williams","author":"Rumelhart David E.","year":"1988","unstructured":"David E. Rumelhart , Geoffrey E. Hinton , and Ronald J . Williams . 1988 . Learning representations by back-propagating errors. In Neurocomputing : Foundations of Research. MIT Press . David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1988. Learning representations by back-propagating errors. In Neurocomputing: Foundations of Research. MIT Press."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10990-008-9037-1"},{"key":"e_1_2_2_54_1","volume-title":"Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus","author":"Spivak Michael","unstructured":"Michael Spivak . 1965. Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus . Addison-Wesley . Michael Spivak. 1965. Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus. Addison-Wesley."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/322169.322183"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/355586.364791"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236765","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236765","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:41:28Z","timestamp":1750282888000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236765"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,30]]},"references-count":55,"journal-issue":{"issue":"ICFP","published-print":{"date-parts":[[2018,7,30]]}},"alternative-id":["10.1145\/3236765"],"URL":"https:\/\/doi.org\/10.1145\/3236765","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,7,30]]},"assertion":[{"value":"2018-07-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}