bendtorch

Releases (6)

• 1.3.0.0 indexed
• 1.2.0.0 unindexed
• 1.1.1.0 unindexed
• 1.1.0.0 unindexed
• 1.0.1.0 unindexed
• 1.0.0.0 unindexed

Definitions (1,049)

dtype.bend

• def fill_f
• def q8_fin
• def Q8.quantizeW quantized by rows (the matrix is consumed)
• type FL
• def read_f
• def fl_list
• def q8_atbyte e of the packed words, as its integer
• type QLthe words and scales, for printing and tests
• def Q8.lists
• def dq_rowW back in F32: q[i, j] · s_i, one F32 product each
• type UF
• def dq_rows
• def dq_fin
• def dq_go
• def Q8.dequantize
• type QD
• def q8_dotΣ_l x[xi + l] · q[qe + l] for l < k, in order, from acc
• type QS
• def q8_put2
• def q8_put
• def q8_colsthe m outputs of row i
• def q8_rowsm
• type Q8Mul
• def q8_mm_fin
• def Q8.matmul_nty = x · Wᵀ; x and W come back
• type DW
• def DW.zero
• def DW.value
• def dw_normhi + lo when |hi| >= |lo| (Fast2Sum), as a renormalized pair
• def dw_finhi + x: TwoSum's s and its exact error, plus the old lo and an extra error term c
• def dw_add3
• def DW.add
• def split_hi
• def two_prod_ea · b exactly as p + e
• def DW.fmaacc + a · b, with the product's rounding error kept
• type AF
• def sum_f
• def sum_dw
• type AF2
• def dot_f
• def dot_dw
• def Sum.f32the sums of the first n numbers of a (the array comes back)
• def Sum.dw
• def Dot.f32
• def Dot.dw
• def bf16_mat
• def cast_f
• def cast_fwd
• def cast_bwd
• def CastOut
• def castp_f2
• def castp_f
• def castp_b2
• def castp_b
• def CastP

main.bend

• type Fwdwhat a forward pass returns: the parameters, the output, what the backward pass will need
• type Bwdwhat a backward pass returns: the parameters, the gradient of the input, the accumulated gradient of the parameters
• type Twoa pair of owned values (parameters of two layers or their saved values)
• def Pair3Type helpers for the left-associated pairs used by Seq/Fork and for Residual's two parameter-free layers.
• def ResidualType
• type PTreeAdamW's moments stored as packed BF16 (CAdamWBF16): a tree that follows the parameters' pairs as the step walks them, with the buffers of each matrix at its leaf.
• type OptStateOwned optimizer buffers: SGD has none, momentum one, Adam/AdamW two (Packed: two, in packed BF16, see PTree).
• def OptState.mapTwo affine maps are supplied separately: each is called at most once.
• def OptState.splitA packed tree splits at its node; a part not yet stepped splits into two such parts.
• def OptState.joinPair the buffers present on both sides.
• type Stepwhat an optimizer step returns: the new parameters, the zeroed gradient, the optimizer state
• type OptThe optimizer, with the scalars of the current step.
• type PLParameters and their flat dump in a fixed order.
• type PALoading cursor: the shared source array and next element offset, handed back without a list or a clone.
• type Layer
• def Layer.fwd
• def Layer.bwd
• def Layer.step
• def Layer.zeros
• def Layer.dumpevery number of the parameters, in the layer's fixed order (the parameters handed back)
• def Layer.loadParameters overwritten from xs at an element offset, in dump order; xs and the next offset come back.
• def Layer.storeThe parameters copied into xs from an element offset, in dump order (load's inverse); xs and the next offset come back.
• def Layer.count
• def ew1_la[t] = f(a[t]) for the n numbers from t; ra holds the array and a[t], already read
• def ew2_la[t] = f(a[t], b[t]): the result replaces a; b comes back unchanged
• def Ew.map
• def ew2_fin
• def Ew.zip(f(a, b), b), element by element
• def f32_add
• def f32_mul
• def Mat.add(a + b, b)
• def zero_of
• def depth_gothe smallest d with 2^d >= n (TA.cap_depth's answer), by doubling a U32: log2(n) cheap steps, where cap_depth compares Nats one unit at a time at each doubling; f is fuel (32 doublings cover every U32)
• def depth
• type Q3
• type Q4
• type MatStep
• def sgd_lw <- w + (-lr)*g, as PyTorch's p.add_(g, alpha=-lr)
• def mom_lbuf <- buf*mu + g; w <- w + (-lr)*buf, as PyTorch's SGD with momentum (no dampening, not Nesterov)
• def adam_gradthe Adam family, in PyTorch's order of operations (torch.optim.adam._single_tensor_adam): g' = g + wd*w (Adam) or w' = w*(1 - lr*wd) (AdamW); m = m + (1-b1)*(g' - m) (lerp); v = v*b2 + (1-b2)*g'*g'; w = w' + (-step*m) / (sqrt(v)/bc2 + eps)
• def adam_decay
• def adam_mc1 = 1 - b1 and c2 = 1 - b2 are given as their own constants (0.1, 0.001), as PyTorch computes 1 - beta in float64 before rounding: 1 - b2 taken in F32 from b2 = 0.999 (0.99900001287 in F32) is 1.3e-5 too small, and over 100 steps that alone put the tiny Llama 21x further from float64 than PyTorch's float32 (NOTES, 2026-10-18)
• def adam_v
• def adam_w
• def adam_l