BendMlBpeTokenizer.Cnt: type of bend-ml-bpe-tokenizer@0.1.3.0 Import: import bend-ml-bpe-tokenizer@0.1.3.0/main.bend as BendMlBpeTokenizer Lines 172-180 of bend-ml-bpe-tokenizer@0.1.3.0/main.bend: https://jend.leeeet.dev/read.txt?path=bend-ml-bpe-tokenizer@0.1.3.0/main.bend&start_line=172&end_line=180 import Base # ===================================================================== # Training (needs no proof: it generates tables; a table is valid if wf(table)) # ===================================================================== # --------------------------------------------------------------- # training: count adjacent pairs, merge the most frequent one, repeat # (tie-break: the first pair that appeared, as in minbpe) # --------------------------------------------------------------- type Cnt is Data: Cnt{a: Tk, b: Tk, n: Nat} Uses (2): https://jend.leeeet.dev/definition.txt?path=Base/base.bend&name=Nat https://jend.leeeet.dev/definition.txt?path=bend-ml-bpe-tokenizer@0.1.3.0/main.bend&name=Tk