bendtorch

Releases (6)

• 1.3.0.0 indexed
• 1.2.0.0 unindexed
• 1.1.1.0 unindexed
• 1.1.0.0 unindexed
• 1.0.1.0 unindexed
• 1.0.0.0 unindexed

Definitions (1,049)

main.bend

• def gnb_grp
• def gnb_groups
• def gnb_images
• def gn_bwd
• def GroupNormGroupNorm over n images of hw pixels and ng groups of cg channels, channels last; its parameters are LayerNorm's
• def emb_cprow i of the output (d numbers) copied from row ids[i] of the table, for the first n ids
• def emb_f2
• def emb_f
• def emb_addg[gi + j] = g[gi + j] + dy[di + j], j = 0..k
• def emb_sceach dY row added into the gradient's row ids[i], in order, by index
• def emb_b2
• def emb_bwd
• def Embeddinga table of v rows of d numbers, looked up by n ids; an id has no gradient, so dX is the empty list
• def lnb_f2
• def lnb_b3
• def lnb_b2
• def LinearNB
• def Id
• def pair_step2one optimizer step on two layers' parameters, of any types
• def pair_step
• def pair_f4
• def pair_f3runs lb on x2 once la's forward pass is done
• def fork_f2
• def fork_b3
• def fork_b2
• def fork_bwd
• def fork_fwd
• def Fork
• def both_b3
• def both_b2
• def both_bwd
• def both_fwd
• def Both
• type Three
• def tie_f4
• def tie_f3
• def tie_f2
• def tie_fwd
• def tie_b4
• def tie_b3
• def tie_b2
• def tie_bwd
• def TieThe shared parameter steps once (through first's step; last's must be the same kind of layer for P, e.g.
• def add2_f
• def Add2
• def mul2_f3
• def mul2_f2
• def mul2_f
• def mpair_a
• def mul2_b2d(a*b) = (dy*b, dy*a)
• def Mul2
• def ResidualOfover any X with a copy (dup), an addition (add) and a layer that adds a pair (join): Seq(Fork(Id, f), join).
• def Residual
• type CS
• def rope_angthe angle of table entry e: inv[c] * position, position e / half, c = e mod half, inv[c] = 1 / base^(2c / (2*half))
• def rope_tabentries e..
• type RS3
• def rope_p3one pair (a = x[i1], b = x[i2]) rotated in place; fwd picks the direction: (a cos - b sin, b cos + a sin) forward, (a cos + b sin, b cos - a sin) backward, in PyTorch's order of operations for each
• def rope_p2
• def rope_onepair e of the n x w matrix: row e / hh (hh = h * half pairs a row), head (e mod hh) / half, column c of it
• def rope_epairs e.., left more after e
• def rope_fin
• def rope_go2
• def rope_go
• def RoPEn rows (b sequences of t positions), w = h heads of 2*half numbers each
• type AttnS
• def kv_ofthe key/value head that query head hi reads, with g query heads per key/value head; the forward and the backward pass both pick it here (the law kv_of_group says which heads share one)
• def rcatxs reversed onto acc
• type DR
• def drowthe dots of one row of a (hd numbers from ia) with consecutive rows of b (hd numbers from ib, each bs further): the current one r and m more, in order
• def dr_row
• def qrowthe scores of the query at position i (row i of q, qs apart, the head's hd columns from qc) against keys 0..i (rows of k, ks apart, from kc): the one place attention reads keys, i + 1 of them (the law causal_keys)
• type QK
• type PW
• def pwxs scaled by sc, written into p from idx on; n counts the numbers written (from the n given)
• type PR
• def pr_n
• def pr_of
• def prow3the softmax over the n numbers pw wrote; the row's count is the one the softmax reports
• def prow2
• def prowrow i of a head's P (t x t, row-major, created zeros): the scaled scores of query i against keys 0..i (qrow) written from the row's start, then the softmax over as many positions as were written: i + 1 (the law causal).
• def prowsrows i..
• def probsP of one head: queries in q (row stride qs, columns from qc), keys in k (row stride ks, columns from kc)
• def frow---- the same P with every key visible: full attention.
• def fprow2
• def fprowrow i of a head's P with every key: the scaled scores of query i against keys 0..t-1 (frow), then the softmax over all t of them (the law full)
• def fprows
• def fprobs
• type PO
• def pmulo[i·ow + oc + d] = a[i·t + 0]·b[0·bs + bc + d] + ...
• def acc_set
• def pacco[j·c + d] += a[0·t + j]·b[0·bs + bc + d] + ...
• def rows_outrows r0..
• def a2_b
• def rows_ina block of t rows, c wide, written into dst from row r0; the block is dropped
• def pmul_copy
• def cols_indst's columns oc..
• def pick_copyt x c numbers of src read from rows r0..
• def block_outthe block of rows r0..
• def block_ina t x c block into dst's rows r0..