bendtorch-gpu

Releases (6)

• 1.3.0.0 indexed
• 1.2.0.0 unindexed
• 1.1.1.0 unindexed
• 1.1.0.0 unindexed
• 1.0.1.0 unindexed
• 1.0.0.0 unindexed

Definitions (324)

main.bend

• def lin_fwd
• def lin_b2
• def lin_bwd
• type EN
• def nn
• def ew1_loopout[i] = f(x[i]) for the leaf's cells
• def ew1
• def ew2_loopout[i] = f(x[i], y[i]) for the leaf's cells
• def ew2
• def ewi_loopo[i] = f(o[i], y[i]) for the leaf's cells, in place (o's old value is the first argument)
• def ewi
• def f32_id
• def lf_copy
• def lf_silu
• def lf_siludo = dy, written in place, x1 = x: main.bend's silu_d(dy, x)
• def lf_addo = a, written in place, x1 = b: a + b
• def lf_mul
• def mulb_loopx1 = dy; the saved a and b, written in place: a becomes db = dy * a and b becomes da = dy * b, in one bang
• def lf_mulb
• def en
• def r1_a
• def dup_fin
• def Mat.duptwo copies of a matrix, the second made on the device
• def Mat.sum2a + b, as main.bend's Mat.sum2 adds each cell, written over a (both are consumed)
• def silu_f2
• def silu_f
• def silu_b
• def SiLUSiLU: y = silu(x), the input saved; dx = silu'(x) * dy
• def mul2_f2
• def mul2_f
• def mul2_b2
• def mul2_b
• def Mul2Mul2: y = a * b, the inputs saved; da = dy * b, db = dy * a
• def add2_f
• def Add2
• def ResidualT.Residual with these copies and sums
• type ESGD
• def sgd_loop
• def lf_sgd
• type EMOM
• def mom_loop
• def lf_mom
• type EADAM
• def adam_loop
• def lf_adam
• def sgd_fin
• def mom_fin
• def opt_fin
• def opt_matone matrix's step, in place: SGD writes two arrays, momentum three, Adam four
• def mat_step
• def lin_step2
• def lin_step
• def LinearF
• def linf_bwdLinearFirst: no dX (the first layer's input needs no gradient)
• def LinearFirstF
• def lnb_b2LinearNB: y = x·Wᵀ, no bias (Llama's projections)
• def LinearNBF
• type ERF
• def r3_y
• def rmsf_if
• def lf_rmsfx1 = x, x2 = the gain g; y a row a leaf
• def erf
• def rms_f2
• def rms_f
• def gdx_w
• def gdx_3
• def gdx_2
• def gdx_if
• def lf_gdxx1 = x, x2 = dy, x3 = g; outputs dx and 1/rms of each row
• def gdg_loopacc + dy[i]·(x[i]·r[row]) down a column, i from j by d
• def gdg_put
• type EDG
• def gdg_if
• def lf_gdgx1 = x, x2 = dy, x3 = 1/rms; the gradient of the gain, in place: a leaf a column
• def rms_b3
• def rms_b2
• def rms_b
• def RMSNormRMSNorm over the last dimension, its gain trained (main.bend's RMSNorm)
• type EGN
• def egn
• def r4_y
• def gnf_if
• def lf_gnfx1 = x, x2 = the gain g, x3 = the bias b; y a block a leaf
• def gn_f2
• def gn_f
• type GB
• def gb_dx
• def gnb_segdx[off + j] = (((dy·g - ma) - xh·mb)·r) along one row's segment
• def gnb_segs
• def gnb_blk3
• def gnb_blk2
• def gnb_blk
• def gnb_if
• def lf_gnbx1 = x, x2 = dy, x3 = g; outputs dx and each block's mean and 1/sqrt(var + eps)
• type GS
• def gdw_rowssg + dy·xh and sb + dy down one image's rows of a channel, i its cell, by c
• def gdw_blk2
• def gdw_blk1
• def gdw_blkblock bl's mean and 1/sqrt(var + eps), then its rows
• def gdw_imgsa channel's images in order: i steps by hw·c to the next image's first row, bl by ng to its block of the same group