bend-ml-llama

Search uses release 0.2.1.0 · 213 definitions

Releases (4)

0.2.1.0 indexed
0.2.0.0 unindexed
0.1.1.0 unindexed
0.1.0.0 unindexed

Definitions (213)

main.bend

def head_att
def group_attthe query heads of one group over its key/value head, in order; the arrays are threaded through
type ATevery group, in order: the outputs of query heads 0, 1, ..., and the caches handed back
def at_join
type GWone key/value head's work: its group of query heads, its keys and its values
def gws
def seq_attsome groups one after another, in order
type GSa list of groups cut after its first n (a tail loop: a helper in the fork tree must not hold the stack)
def cut
def halvingshow many times n groups are halved (the larger half kept) until one or none is left
def par_attthe n groups of gs (cut in two halves) in a tree of parallel lets of depth d, each leaf a half of a node; the outputs and the caches come back in group order, so every head computes what it computed one after another
type KAthe attention output and the cache handed back
def ka_of
def KA.then
def attentionattention of the (rotated) query heads at position t over the cache of positions 0..t (the newest included).
type LWthe weights of one layer, with their shapes from the config (row = output, as PyTorch stores them)
type Actwhat a layer computes at one position, every intermediate: the tests compare them with transformers, and a backward pass would read them
type BRthe layer's output at position t, its cache grown to t + 1 positions, its weights handed back, and the activations
def BR.thencontinues with the fields of a layer's result (a computed value cannot be matched; a loop over positions recurses inside k)
def mv.go
def mvy = W · x, then `then` with W handed back and y (an Array has one owner, so a product returns its matrix)
def blockthe layer at position t (0-based) on input x, with the cache of positions 0..t-1 and the rotation of t; d is the parallel depth of attention (see attention)
def block_genblock for generation: the same computation, with the activations left empty.
def block_kkeep: block with the activations (the tests read them), or block_gen without them; chosen before anything is computed
type Modelemb: the embedding (vocab rows; also the LM head, tied); lnf: the final norm's gains
type Runthe last layer's output, the layers and caches handed back, and every layer's activations
def runpend is the previous layer's result: its output is the next layer's input
def layers
type Outwhat one position produces: the logits, the final norm, every layer's activations, the caches grown to t + 1 positions and the model handed back
def Out.thencontinues with the fields of a position's result: logits, final norm, activations, caches, model
def head_ofthe final norm, then logits = emb · nf (the LM head is the embedding matrix)
def forward.go
def forward_ktoken tok at position t (0-based), with the caches of positions 0..t-1 (one per layer) and room for cap positions; room is the proof that t < cap; keep: whether the result carries every layer's activations
def forwarda position with every layer's activations (the tests read them)
def forward_gena position for generation: the same logits, caches and model.
def pre_ofa prompt position, whose logits nobody reads: the layers and the caches as forward_gen gives them, without the final norm and the product by the embedding (vocab x dim, the largest of a position), so lg and nf are empty
def forward_pre.go
def forward_pre
def cachesn empty caches, one per layer, each with room for cap positions
type AMthe first index of the largest value (torch.argmax does the same), and the value
def AM.i
def AM.v
def gt_head
def argmax.go
def argmax
def hash32Chris Wellons' lowbias32 hash (public domain): xor-shifts and two odd multipliers, a bijection on U32
def uniformdraw n of a seed, a uniform F32 in [0, 1): the top 24 bits of hash32(hash32(seed) + n), times 2^-24 (exact).
type Canda candidate: a logit and its id
def Cand.i
def Cand.v
def goes_beforewhether a value v goes before the head of cs: the head is smaller (an equal one keeps its place, ahead of v)
def insv (id i) into the candidates cs, largest first; g: whether it goes before the head
def last_vthe value of the last candidate (lo if there is none)
def enterswhether the head of xs enters the candidates: there is room, or it is above the smallest one (lo)
def topk.gothe k largest of xs (the values from id i on) and of the candidates cs, largest first; cs holds m <= k of them, lo the smallest; g: whether the head of xs enters.
def topk
def weightthe weight of a logit v: exp(v/temp - top/temp), top the largest logit (its weight is 1)
def totalacc plus the weights of cs, in order
type Kepthow many candidates top-p keeps (n) and their weight (z)
def kept.gog: whether the head of cs is kept (the weight acc before it is below lim = p·Z; the first is always kept)
def stopswhether the draw stops at the head of cs: it is the last kept one (q = 0 kept after it), or the cumulative weight acc plus its own is above the target u·Zp
def pick.gothe candidate drawn; s: whether it is the head of cs
def sample.pick
def sample.cs
def top_of
def samplea token drawn from the logits xs (its id and logit) with temperature temp > 0, the k > 0 largest logits and top-p p, by the uniform draw u
def belowevery candidate's id is below n
def and_lthe two halves of a true Bool.and
def and_r
law ins_belowPROOF: by induction on cs.
law take_belowPROOF: taking the first k candidates keeps a part of them.
law topk_belowPROOF: by induction on xs, with i + length(xs) = n: the head of xs has id i < n (step_room), and after it 1 + i + length(tail) = n still holds (step_inv); ins_below and take_below keep the candidates' ids below n.
law pick_belowPROOF: by induction on cs: the draw returns a candidate of cs, whose id is below n, or id 0 < n when cs is empty.
def sample.pick_below
law sample_belowLAW (sample_below): from logits xs with at least one value, the id sample draws is below the length of xs: an id of the vocabulary, whatever the temperature, k, p and draw.
def bitcast4 little-endian bytes -> F32 (U32 and F32 hold the same Word(32n): it is a bitcast)
def f32_le
def f32s
def rf_loopreads the file in blocks of `blk` bytes; k blocks remaining; m is the block already read
def blocks_of
def read_floatsn F32 numbers (little-endian) from a file (the tests' inputs; the weights are BF16, below)
def depth_selthe weight matrices are kept in 2^d row bands (TA.Bands); each matrix · vector product runs its bands in parallel without copying the weights (bend-ml NOTES exp.
def depth_for
def bf16_le2 little-endian bytes of a BF16 number -> the F32 whose upper 16 bits they are (exact)
def bf16s
def fill_bf16writes the BF16 numbers held in bs (2 bytes each), widened to F32, at index i, without building a list of F32 first
def fill_bf16_mat
def stream_exactreads blocks of at most 1 MB (2^19 numbers) until rem bytes have been read, so a band never takes numbers that belong to the next one
def get_bandrr x c numbers from number `start` of the tensor at byte lo, into a band of r rows (rr == r, see load_tree; the Array is sized for rr * c numbers as Mat.zeros would size it)
def load_treefills a tree of 2^d bands (TA.Bands.zeros's shape: each node halves its rows with TA.half) in row order; rr is r at run time.
def span_cmp
def span_ok
def check_spanstops the program unless tensor t, n BF16 numbers from byte hi * 2^32 + lo, ends inside the file with hi = 0
def rb_done
def read_bytes
def digit_head
def u32s.gothe decimal numbers in bs, separated by anything that is not a digit; dg says whether bs starts with a digit
def u32s
def cfg_of
def read_cfgthe config of the model in dir (written by reference/prep.py)