TransformerBlock constructor

TransformerBlock(
  1. int dModel,
  2. int nHeads, {
  3. int ffnMult = 4,
  4. bool rope = false,
})

Implementation

TransformerBlock(int dModel, int nHeads, {int ffnMult = 4, bool rope = false})
    : ln1 = LayerNorm(dModel),
      ln2 = LayerNorm(dModel),
      mha = MultiHeadAttention(dModel, nHeads, rope: rope ? const RoPE(RoPEConfig()) : null),
      ffn = FeedForward(dModel, hiddenMult: ffnMult);