Expert class

A single expert. Either a two-layer MLP (variant = mlp, default) or a SwiGLU gated FFN (variant = swiGlu), matching DeepSeek-V3 and Mixtral.

Inheritance

Constructors

Expert(int dim, int hiddenDim, {Device device = Device.CPU, int seed = 0, ExpertActivation activation = ExpertActivation.relu, ExpertVariant variant = ExpertVariant.mlp})

activation → ExpertActivation: final
dim → int: final
hashCode → int: The hash code for this object.
no setterinherited
hiddenDim → int: final
runtimeType → Type: A representation of the runtime type of the object.
no setterinherited
training ↔ bool: Whether this module is in training mode. Layers that behave differently between training and inference (e.g. Dropout) read this flag in their call method. Defaults to training mode.
getter/setter pairinherited
variant → ExpertVariant: final
w1 → Linear: final
w2 → Linear: final
w3 → Linear?: Gate projection [dim, hiddenDim], only allocated when variant is ExpertVariant.swiGlu.
final

call(Tensor x) → Tensor
eval() → void: Put this module (and any registered submodules) into evaluation mode.
inherited
noSuchMethod(Invocation invocation) → dynamic: Invoked when a nonexistent method or property is accessed.
inherited
parameters() → List<Tensor>: Trainable tensors owned by this module (and its submodules).
override
submodules() → List<Module>: Submodules owned by this module. Subclasses that compose other modules should override this so train() / eval() propagate. Default: empty.
override
toString() → String: A string representation of this object.
inherited
train() → void: Put this module (and any registered submodules) into training mode.
inherited
zeroGrad() → void: Zero every parameter's gradient. Safe to call before each backward.
inherited