bendtorch

Releases (6)

• 1.3.0.0 indexed
• 1.2.0.0 unindexed
• 1.1.1.0 unindexed
• 1.1.0.0 unindexed
• 1.0.1.0 unindexed
• 1.0.0.0 unindexed

Definitions (1,049)

main.bend

• def sm_fwd
• def smb_fixdy[j] = y[j] * (dy[j] - s), j = idx..idx + k; rd and ry hold dy[idx] and y[idx]
• def smb_row
• def smb_rowssoftmax backward, as PyTorch's: dx = y * (dy - sum(dy * y)), row after row, in place over dy
• def dx_fin
• def sm_bwd
• def Softmaxsoftmax over each row; saves its output
• def lsumacc + exp(a[j] - m) for j = idx..idx + k, in order
• def lwa[j] = (a[j] - m) - l for j = idx..idx + k
• def lsm_r3
• def lsm_r2
• def lsm_row
• def lsmf_rows
• def lsm_fwd
• def asumacc + a[j] for j = idx..idx + k, in order
• def lsmb_fixdy[j] = dy[j] - exp(y[j]) * s, j = idx..idx + k
• def lsmb_row
• def lsmb_rowsdx = dy - exp(y) * sum(dy), row after row, in place over dy
• def lsm_bwd
• def LogSoftmax
• def mat_step_finone optimizer step on a layer whose parameter is a single matrix
• def mat_step
• def g_add_fin
• def Mat.sum2a + b, dropping b
• def dup_fin
• def Mat.duptwo copies of a matrix
• def ml_list
• def ssqacc + a[i]·a[i] + ...
• def rms_inv1 / rms of a row whose sum of squares is s
• type R3
• def rms_wy[off + j] = g[j] * (x[off + j] * r), j = jj..jj + k; rx and rg hold x[off + jj] and g[jj]
• def rmsf_row2
• def rmsf_row
• def rmsf_rowsrows i..
• def rms_f2
• def rms_fwd
• type R5with r = 1/rms, xh = x*r, dxh = dy*g: dx = r * (dxh - xh * mean(dxh * xh)); dg += dy * xh, row after row
• type R3S
• def rmsb_dotacc + dxh[j]·xh[j] + ..., in order; rx, rd, rg hold x[off + jj], dy[off + jj], g[jj]
• def dg_add
• def rmsb_wdx[off + j] = ((dy[j]·g[j]) - (x[j]·r)·m)·r and dg[j] += dy[j]·(x[j]·r), j = jj..jj + k
• def rmsb_row3
• def rmsb_row2
• def rmsb_row
• def rmsb_rows
• def gain_bwd_fin
• def rms_b2
• def rms_bwd
• def RMSNormRMSNorm over rows of d numbers, with a learned gain (Llama's norm); saves its input
• type LN
• def csqacc + (a[j] - mu)·(a[j] - mu) for j = idx..idx + k, in order
• type ASM
• def ln_mr2
• def ln_mr1
• def ln_mra row's mean and 1 / sqrt(var + eps), as ln_xh and ln_r computed them
• type R4
• def lnf_wy[off + j] = ((x - mu) * r) * g[j] + b[j], j = jj..jj + k
• def lnf_row2
• def lnf_row
• def lnf_rows
• def ln_f2
• def ln_fwd
• type R3SSwith xh = (x - mu) * r and dxh = dy * g: dx = r * ((dxh - mean(dxh)) - xh * mean(dxh * xh)); dg += dy * xh; db += dy
• def lnb_sumssa + dxh[j] and sb + dxh[j]·xh[j], j = jj..jj + k, each in order
• def ln_dw
• type R4B
• def lnb_wdx[off + j] = (((dxh - ma) - xh * mb) * r), dg[j] += dy * xh, db[j] += dy, j = jj..jj + k (ma, mb the two means)
• def lnb_row3
• def lnb_row2
• def lnb_row
• def lnb_rows
• def ln_b2
• def ln_bwd
• def ln_pair
• def ln_unpair
• def ln_s2
• def ln_step2
• def ln_step
• def ln_dump2
• def ln_dump
• def ln_load3
• def ln_load2
• def ln_load
• def ln_store2
• def ln_store
• def LayerNorm
• def gn_sumacc + the segments of the block's remaining `left` rows, each one row (c cells) below the last, in order
• def gn_csqacc + (x - mu)·(x - mu) over the same segments
• def gn_mr1
• def gn_mra block's mean and 1 / sqrt(var + eps); off is its first cell, nf = hw·cg
• def gnf_rowsy over the block's rows, one segment each (LayerNorm's lnf_w with the group's gains)
• def gnf_grp2
• def gnf_grpone (image, group) block: off is the image's first row's first cell, j the group's first channel
• def gnf_groups
• def gnf_images
• def gn_fwd
• def gnb_sumsthe backward pass is LayerNorm's with the block in place of the row: the two means (of dxh and of dxh·xh) are taken over the whole block, then each cell's dx, dg[ch] += dy·xh and db[ch] += dy, in order
• def gnb_rows
• def gnb_grp3
• def gnb_grp2