Activation.swish constructor

Activation.swish()

Swish activation function with beta = 1

swish(x) = x*sigmoid(x)

In loredart swish is equivalent to SiLU

Example:

final swish = Activation.swish();
final x = Matrix.row([-1, 0, 1]);
final y = swish.function(x);
print(y); // output: matrix 1⨯3 [[-0.2689414213699951, 0.0, 0.7310585786300049]]

Implementation

Activation.swish() {
  function = (Matrix m, [dynamic param]) =>
      m.apply((double x) => x / (1 + math.exp(-x)));
  dfunction = (Matrix m, [dynamic param]) {
    final swishMatrix = function(m);
    return [
      swishMatrix +
          (m.apply((double x) => 1 / (1 + math.exp(-x))) %
              (-swishMatrix.addedScalar(-1)))
    ];
  };
  name = 'swish';
}