law smollm2_135m_sizes

bend-ml-llama@0.2.0.0 · main.bend:161

import Base

161# LAW (the configs): SmolLM2-135M has 9 query heads of 64, a hidden size of 576 and keys of 192; SmolLM2-360M
162# has 15 query heads of 64, a hidden size of 960 and keys of 320; SmolLM2-1.7B has 32 query heads of 64, a hidden
163# size of 2048 and keys of 2048 (as many key/value heads as query heads), as their config.json files say.
164law smollm2_135m_sizes:
165  {Cfg.dim(smollm2_135m()) == 576n : Nat} & {Cfg.heads(smollm2_135m()) == 9n : Nat} & {Cfg.head(smollm2_135m()) == 64n : Nat} & {Cfg.kvw(smollm2_135m()) == 192n : Nat}