bendtorch

Releases (6)

• 1.3.0.0 indexed
• 1.2.0.0 unindexed
• 1.1.1.0 unindexed
• 1.1.0.0 unindexed
• 1.0.1.0 unindexed
• 1.0.0.0 unindexed

Definitions (1,049)

main.bend

• def seq_at_load2
• def seq_at_load
• def seq_at_store
• def Seq.at
• type LossOut
• type TrainParameters, gradient (zero between steps), optimizer-specific buffers, Adam's 1 - b1^t and 1 - b2^t, last loss.
• def OptState.new
• def q_start1 - b^t starts at 0; a configuration OptCfg.ok rejects starts at NaN, so every loss after the first is NaN.
• def Train.new
• def ts5
• def ts4
• def ts3
• def ts2
• def Train.stepforward, loss, backward and one optimizer step: the same code for every model
• def train_at_bOne owned model, including dimensions captured at run time.
• def train_at_loss
• def train_at_f
• def Train.step_at
• def bitcastU32 and F32 hold the same Word(32n): these are bitcasts
• def bits
• def f32_le
• def f32s
• def rd_done
• def read_atn bytes from offset off
• def mat_or_die
• def Mat.loadan r x c matrix stored as r*c little-endian F32 from byte off of a file
• def words
• def ml_words
• def Mat.printone line: the name, then each number as the decimal of its 32 bits (exact, for the tests in reference/)
• def F32.print
• def zeros_an zeros, in an array with room for them
• def acti_f3an activation whose derivative needs its input (silu, gelu): saves a copy of the input
• def ActIn
• def sigmoid_v
• def sigmoid_dPyTorch's sigmoid_backward: dy * (1 - y) * y
• def Sigmoid
• def tanh_v
• def tanh_d
• def Tanh
• def silu_vsilu(x) = x * sigmoid(x), computed as PyTorch's CPU kernel: x / (1 + exp(-x)); its gradient dy * s * (1 + x * (1 - s)) with s = sigmoid(x)
• def silu_d
• def SiLU
• def gelu_innerGELU in its tanh form (PyTorch's approximate="tanh"): 0.5 x (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))).
• def gelu_v
• def gelu_d
• def GELU
• def erf_hpone Horner step, c + t s; erf_hm is the step for a negative coefficient -c: t s - c gives the bits of -c + t s
• def erf_hm
• def erf_small
• def erf_p0[0.84375, 1.25): t = a - 1.0
• def erf_p1[1.25, 1.75): t = a - 1.5
• def erf_p2[1.75, 2.25): t = a - 2.0
• def erf_p3[2.25, 2.75): t = a - 2.5
• def erf_p4[2.75, 3.25): t = a - 3.0
• def erf_p5[3.25, 3.92): t = a - 3.5
• def erf_sign
• def erf_i0the pieces are chosen from the top down, so that only one polynomial is computed (each test is a match on a Bool parameter); b is "a is at least the piece's lower end"
• def erf_i1
• def erf_i2
• def erf_i3
• def erf_i4
• def erf_i5
• def erf_top
• def erf
• def gelu_erf_cdfGELU in its exact form, PyTorch's default (approximate="none"): gelu(x) = x/2 (1 + erf(x / sqrt 2)), in PyTorch's CPU order, (x * 0.5) * (1 + erf(x * 0.70710677)).
• def gelu_erf_v
• def gelu_erf_d
• def GELUErf
• type DropP
• def mix32lowbias32 (Chris Wellons): an invertible mix of the 32 bits
• def drop_kthe factor of number i: 1 / (1 - p) if its uniform u = (h >> 8) / 2^24 is >= p, else 0
• def drop_l
• def drop_mask
• def drop_f2
• def drop_fwd
• def drop_b2
• def drop_step
• def drop_dump
• def drop_load_fin
• def drop_load
• def drop_storethe counter, bitcast to one F32 as dump writes it
• def Dropout
• type D2
• type A2
• def sdota[ia]·b[ib] + a[ia + sa]·b[ib + sb] + ...
• def dot0
• type AS
• def pmaxthe largest of a[idx] (held in r) and the k numbers after it: max(max(a0, a1), a2) ..., from the first
• def pmax0
• def pexpa[j] = exp(a[j] - m) for j = idx..idx + k, their sum added to acc in order
• type PCthe array and how many numbers the last loop of a softmax divided
• def pc_a
• def pdiva[j] = a[j] / s for j = idx..idx + k, counted onto c
• def psm3
• def psm2
• def psmcsoftmax in place over the n numbers of p from off (n >= 1), as PyTorch: exp(x - max) / sum; with the count of the numbers it divided (the law causal reads it)
• def psm
• def copy_ndst[di..] = src[si..], cnt + 1 numbers; r holds src[si], already read
• def smf_rowsrows i..
• def out_savedthe output, and a copy of it saved for the backward pass