law smollm2_135m_sizes

bend-ml-llama@0.2.1.0 · main.bend:162

import Base

162# LAW (the configs): SmolLM2-135M has 9 query heads of 64, a hidden size of 576 and keys of 192; SmolLM2-360M
163# has 15 query heads of 64, a hidden size of 960 and keys of 320; SmolLM2-1.7B has 32 query heads of 64, a hidden
164# size of 2048 and keys of 2048 (as many key/value heads as query heads), as their config.json files say.
165law smollm2_135m_sizes:
166  {Cfg.dim(smollm2_135m()) == 576n : Nat} & {Cfg.heads(smollm2_135m()) == 9n : Nat} & {Cfg.head(smollm2_135m()) == 64n : Nat} & {Cfg.kvw(smollm2_135m()) == 192n : Nat}