SingleHeadAttention class

Implements a single head of the self-attention mechanism.

This layer learns the contextual relationships between tokens in a sequence by calculating Query, Key, and Value projections and producing a weighted average of the values based on the query-key similarity. Implements a single head of the self-attention mechanism.

This layer is the core of the Transformer architecture. It learns contextual relationships between tokens in a sequence. It does this by creating Query (Q), Key (K), and Value (V) projections for each input token and then producing a weighted average of the Values based on the Query-Key similarity.

The attention formula is: $Attention(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V$

• Input: A Tensor<Matrix> of shape [sequence_length, dModel].
• Output: A Tensor<Matrix> of shape [sequence_length, dV]. Implements a single head of the self-attention mechanism. Implements a single head of the self-attention mechanism.

Inheritance

• Object
• Layer
• SingleHeadAttention

Constructors

SingleHeadAttention(int dModel, {int? dK, int? dV})

Properties

dK ↔ int

getter/setter pair

dModel ↔ int

getter/setter pair

dV ↔ int

getter/setter pair

hashCode → int

The hash code for this object.

no setter, inherited

lastAttentionWeights ↔ Tensor<Matrix>

Stores the attention weights from the most recent forward pass. Useful for debugging and visualization.

getter/setter pair

name ↔ String

A user-friendly name for the layer (e.g., 'dense', 'lstm').

getter/setter pair, override-getter

parameters → List<Tensor>

A list of all trainable tensors (weights and biases) in the layer.

no setter, override

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

Wk ↔ Tensor<Matrix>

getter/setter pair

Wq ↔ Tensor<Matrix>

getter/setter pair

Wv ↔ Tensor<Matrix>

getter/setter pair

Methods

build(Tensor input) → void

Initializes the layer's parameters based on the shape of the first input.

override

call(Tensor input) → Tensor

The public, callable interface for the layer.

inherited

forward(Tensor input) → Tensor<Matrix>

The forward pass now correctly returns only a Tensor

override

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited