law attn_par_fwd

bendtorch@1.3.0.0 · LAWS.bend:189

import Base
import bendtorch@1.3.0.0/main.bend as T

189# LAW (attn_par_fwd): T.Attention runs its sequences in a tree of parallel lets (T.fpar: the sequences cut in halves,
190# each half cut again, d levels deep, the leaves' outputs appended). Whatever the depth and however the sequences are
191# split between the two halves, the tree gives exactly what T.fleaf gives running them one after another, in order: the
192# parallel forward pass is the sequential one, and its output does not depend on d or on the thread count.
193law attn_par_fwd:
194  for d: Nat
195  for gs: T.Halves<T.FS>
196  for +n: Nat
197  for +h: Nat
198  for +dm: T.Dm
199  {T.fpar(d, gs, n, h, dm) == T.fleaf(T.hcat(T.FS, gs), h, dm, Nil{}) : List<&1, Array<F32>>}