type CSM

bendtorch@1.3.0.0 · main.bend:3164

import Base

3164# Cross-entropy's softmax over a row of c classes: as psm, but the row's exponentials are summed as a balanced tree
3165# (tree_sum, on a copy), since a running F32 sum of SmolLM2's 49152 classes was 3.4 tolerances off PyTorch's loss
3166# (NOTES, M10). psm itself keeps its order: attention's rows are short and the law causal reads its count.
3167# a row's probabilities, its max m and log of its sum of exp(x - m) (log_softmax's pieces)
3168type CSM is Type:
3169  CSM{a: Array<F32>, m: F32, ls: F32}