core/nn/transformer library
Pre-LayerNorm Transformer encoder block.
Layout (per Vaswani et al. with pre-norm; more stable to train):
h = x + dropout(mha(ln1(x)))
y = h + dropout(ffn(ln2(h)))
Where ffn(z) = W2(activation(W1(z))) — the classic 2-layer
feed-forward with an inner width of ffnDim (defaults to
4 * embedDim). The activation defaults to ReLU; pass
activation: Activation.geluTanh for the GPT-2 tanh-approx GELU.
Classes
Enums
- Activation
- Activation function for the FFN of a TransformerBlock.