bend-ml-autograd

Releases (2)

• 0.1.1.0 indexed
• 0.1.0.0 unindexed

Definitions (33)

main.bend

• type NEExpressions with one variable NX, over Nat (a commutative semiring).
• def nvalvalue of the expression at x
• def nfwdFORWARD-mode derivative (dual numbers): derivative of the sum and the product rule
• def nbwdREVERSE-mode derivative: g is the gradient arriving from the output; each node passes on g (sum) or g * value-of-the-other-factor (product) to its children and adds up what arrives at NX.
• def ndist_la * (b + c) == a*b + a*c (left distributivity; nat-lemmas only has the right one)
• def nmul_caseThe product case, with everything abstracted into numbers: (g*vb)*fa + (g*va)*fb == g * (fa*vb + va*fb)
• def nbwd_okThe reverse mode returns g times the forward-mode derivative.
• law reverse_eq_forwardLAW (reverse_eq_forward): for any expression made of constants, X, sums and products, the reverse-mode gradient (starting with gradient 1 at the output) equals the forward-mode derivative.
• type G
• def lookup
• def gval
• def acc_addadds v to the gradient of variable i
• def mask2
• def relu_dlocal derivative of relu: 1 if the input > 0, else 0
• def tanh_d
• def gbwdg is the gradient arriving at the node; acc accumulates the gradient of each variable
• def ggradgradient of the expression with respect to each variable of env
• def lineary = x·W + b x: n x i, W: i x o, b: o -> y: n x o
• def linear_bwdGiven the gradient dy arriving at y, returns (dx, (dW, db)): dx = dy · Wᵀ (n x o)·(o x i) = n x i dW = xᵀ · dy (i x n)·(n x o) = i x o db = sum of the rows of dy If any of these products had the wrong dimensions, the program would not compile.
• def mask.l
• def mask.rows
• def relu_bwdy = relu(x); the gradient passes where x > 0
• def onehot.cellone-hot: n rows of c numbers, with 1 at the label's position
• def onehot.row
• def onehot.rows
• def log.l
• def log.rows
• def rows.total
• def ce_loss2
• def ce_lossmean cross-entropy loss: -(1/n) Σ log p[i][label_i]
• def ce_gradgradient of the loss with respect to the logits: (softmax - one-hot) / n
• def sgd
• def sgd_vec