bendtorch

Releases (6)

• 1.3.0.0 indexed
• 1.2.0.0 unindexed
• 1.1.1.0 unindexed
• 1.1.0.0 unindexed
• 1.0.1.0 unindexed
• 1.0.0.0 unindexed

Definitions (1,049)

main.bend

• type FS
• type Dm
• type HCthe columns query head hi reads: its own hd columns of q (hc) and those of its key/value head kv_of(hi, g) in k and v (kc); both passes take their columns from here (the law kv_cols)
• def head_cols
• def hc_kc
• def fs_fin
• def head_f2
• def head_fwdone head's forward pass at the columns hc (its queries) and kc (its key/value head) that head_cols gives; it has no head index, so it reads no other columns
• def fheads
• def fs_o
• type BS
• def gfixg[j] = (p[j] * (g[j] - s)) * sc for j = idx..idx + k; rg and rp hold g[idx] and p[idx]
• def g_set2
• type GW
• type GN
• def gwxs written into g from idx, counted onto n
• type GR
• def gr_n
• def gr3the n positions of the row written: s = sum(dP * P) over them (from +0.0, in order), then G = (P * (dP - s)) * scale
• def gr2
• def growrow i of G (from i·t, t wide; softmax's backward pass, then the scale): dP's row against values 0..i only (drow, as qrow reads keys 0..i), written from the row's start, and s and G over as many positions as were written: i + 1 (the law causal_grad).
• def growsrows i..
• def hb5
• def hb4dK += Gᵀ·Q, then dV += Pᵀ·dO
• def hb3dQ = G·K into the head's columns, then dK
• def hb2
• def hb1P again, then G
• def head_bwdone head's backward pass at head_cols' columns (no head index, as head_fwd)
• def bheads
• type BG
• def bg_finthe group's summed dK and dV into key/value head j's columns of dk and dv
• def bgroup
• def bgroups
• type Halves
• def hcatthe two halves back as one list, in order (what the laws attn_par_fwd and attn_par_bwd compare the trees with)
• def cutxs cut after its first n (a tail loop: a helper in the fork tree must not hold the stack)
• type FL
• def fu3a unit of the parallel work is one sequence's key/value group j: its g query heads' columns of Q (t x g·hd), head j's columns of K and V (t x hd), and an output of its own (t x g·hd).
• def fu2
• def fu1
• def funit
• def fgroups
• def fslicesevery sequence's units, sequence by sequence and group by group, newest first
• def fleaf
• def fparthe n sequences of gs (cut in two halves) in a tree of parallel lets of depth d; the outputs come back in order
• type UO
• def ub_rowone sequence's units (t x gw each) into o's rows r0..
• def blocks_inthe units' outputs into o (n x w), in the order fslices made them
• type BL
• def bu4the backward pass's unit: one sequence's group j, as fslices' (Q, K, V, and dO's columns of the g heads), with dQ, dK and dV of its own
• def bu3
• def bu2
• def bu1
• def bunit
• def bgroups_u
• def bslices
• type DB
• def db_of
• def bleaf
• def bpar
• type D3
• type UD
• def ud_rowone sequence's units into rows r0..: dQ at columns j·gw (w wide), dK and dV at j·hd (kvw wide)
• def dblocks_in
• type AttnDims
• def dm_ofa unit's dimensions: g query heads (g·hd columns) over one key/value head (hd columns)
• def attn_treewhether the units (a sequence's key/value group each, fslices) run in the tree: from 8 units on, or from 2 when each is big.
• def fgo
• def bgo
• def af2
• def attn_fwd
• def head_full---- the same forward with every key visible.
• def fheads_full
• def fleaf_full
• def fpar_full
• def fgo_full
• def af2_full
• def attn_fwd_full
• def ab3
• def ab2
• def attn_bwd
• def fgrow---- the backward of the full-visible forward.
• def fgrows
• def fhb1
• def head_bwd_full
• def bheads_full
• def bgroup_full
• def bgroups_full
• def bleaf_full
• def bpar_full
• def bgo_full
• def ab2_full
• def attn_bwd_full
• def FullAttentionthe same layer as Attention with every key visible; Attention itself is untouched, so the causal laws and the bits of every existing model stay as they are
• def Attentionthe core of attention, after the projections and RoPE; the erased arguments are the shape facts it relies on
• def hit
• def ts_addthe sum of a[0] ..
• def ts_pair
• def ts_pairs
• def ts_put