Activation enum

Activation function for the FFN of a TransformerBlock.

• relu — plain ReLU (default; matches the original Vaswani recipe used by TransformerLM).
• geluTanh — GPT-2 style GELU with the tanh approximation: 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3))).
• quickGelu — OpenAI CLIP's fast GELU approximation: x * sigmoid(1.702 * x). Used by every CLIP vision / text transformer, hence needed to load HuggingFace openai/clip-* checkpoints without numerical drift.

Inheritance

• Object
• Enum
• Activation

Available extensions

• EnumName

Values

relu → const Activation

geluTanh → const Activation

quickGelu → const Activation

Properties

hashCode → int

The hash code for this object.

no setter, inherited

index → int

A numeric identifier for the enumerated value.

no setter, inherited

name → String

Available on Enum, provided by the EnumName extension

The name of the enum value.

no setter

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

Methods

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited

Constants

values → const List<Activation>

A constant List of the values in this enum, in order of their declaration.