{"id":1441,"date":"2017-09-30T12:36:00","date_gmt":"2017-09-30T12:36:00","guid":{"rendered":"https:\/\/nag.com\/?post_type=insights&p=1203"},"modified":"2023-07-11T16:08:02","modified_gmt":"2023-07-11T16:08:02","slug":"automatic-differentiation-in-more-depth","status":"publish","type":"insights","link":"https:\/\/nag.com\/insights\/automatic-differentiation-in-more-depth\/","title":{"rendered":"Automatic Differentiation in More Depth"},"content":{"rendered":"
\n
\n
\n

$\\newcommand{\\R}{\\mathbb{R}} \\newcommand{\\costR}{\\mathcal{R}}$
To delve a little more into AD, consider a computer implementation of a function $f$ with $n$ inputs and one output, i.e. $f:\\R^n\\rightarrow \\R$. AD can be applied to vector-valued functions as well, but to keep things simple we only consider real valued functions below. AD comes in two modes, forward and reverse.<\/p>\n

Forward (tangent-linear) Mode AD<\/h3>\n

The forward (or tangent-linear) AD version of $f$ is a function $F^{(1)}:\\R^{2n}\\rightarrow\\R$ given by<\/p>\n <\/div>\n <\/div>\n<\/div>\n\n\n

<\/div>\n\n\n\n \n y<\/mi>\n \n (<\/mo>\n 1<\/mn>\n )<\/mo>\n <\/mrow>\n <\/msup>\n =<\/mo>\n \n F<\/mi>\n \n (<\/mo>\n 1<\/mn>\n )<\/mo>\n <\/mrow>\n <\/msup>\n (<\/mo>\n \n x<\/mi>\n <\/mrow>\n ,<\/mo>\n \n \n x<\/mi>\n \n (<\/mo>\n 1<\/mn>\n )<\/mo>\n <\/mrow>\n <\/msup>\n <\/mrow>\n )<\/mo>\n =<\/mo>\n ∇<\/mi>\n f<\/mi>\n (<\/mo>\n \n x<\/mi>\n <\/mrow>\n )<\/mo>\n ⋅<\/mo>\n \n \n x<\/mi>\n \n (<\/mo>\n 1<\/mn>\n )<\/mo>\n <\/mrow>\n <\/msup>\n <\/mrow>\n =<\/mo>\n \n (<\/mo>\n \n \n ∂<\/mi>\n f<\/mi>\n <\/mrow>\n \n ∂<\/mi>\n \n x<\/mi>\n <\/mrow>\n <\/mrow>\n <\/mfrac>\n )<\/mo>\n <\/mrow>\n ⋅<\/mo>\n \n \n x<\/mi>\n \n (<\/mo>\n 1<\/mn>\n )<\/mo>\n <\/mrow>\n <\/msup>\n <\/mrow>\n<\/math>\n\n\n\n
<\/div>\n\n
\n
\n
\n

for inputs $\\mathbf{x},\\mathbf{x^{(1)}}\\in\\R^n$ where the dot is regular dot product. To get the whole gradient of $f$ we let $\\mathbf{x^{(1)}}$ range over Cartesian basis vectors and call $F^{(1)}$ repeatedly.<\/p>\n