TransformerBlock constructor

TransformerBlock(
  1. int dModel,
  2. int nHeads, {
  3. int ffnMult = 4,
  4. bool rope = false,
})

Creates a TransformerBlock.

dModel is the model dimension. nHeads is the number of attention heads. ffnMult is the multiplier for the hidden size in the feed-forward network. rope enables RoPE positional encoding if true.

Implementation

TransformerBlock(int dModel, int nHeads, {int ffnMult = 4, bool rope = false})
    : ln1 = LayerNorm(dModel),
      ln2 = LayerNorm(dModel),
      mha = MultiHeadAttention(dModel, nHeads, rope: rope ? const RoPE(RoPEConfig()) : null),
      ffn = FeedForward(dModel, hiddenMult: ffnMult);