Activation.softmax constructor

Activation.softmax()

Softmax activation function

softmax(x) = {e^x_i/sum(e^x) | i = 1,n}

Softmax calculate 'probabilities' from input vector x

Softmax is a vector to vector function, that's why it`s derivative is a Jacobi matrix

Example:

final softmax = Activation.softmax();
final x = Matrix.row([-1, 0, 1]);
final y = softmax.function(x);
print(y); // output: matrix 1⨯3 [[0.09003057317038045, 0.2447284710547976, 0.6652409557748218]]

Implementation

Activation.softmax() {
  function = (Matrix m, [dynamic param]) {
    Matrix resultMatrix = Matrix.zero(n: 0, m: 0);
    for (int i = 0; i < m.m; i += 1) {
      Matrix exps = m.getColumn(i);
      exps = exps.addedScalar(max(exps)).apply(math.exp);
      if (i == 0) {
        resultMatrix = exps.scaled(1 /
            exps.flattenList().reduce((value, element) => value + element));
      } else {
        resultMatrix = MatrixOperation.columnBind(
            resultMatrix,
            exps.scaled(1 /
                exps
                    .flattenList()
                    .reduce((value, element) => value + element)));
      }
    }
    return resultMatrix;
  };
  dfunction = (Matrix m, [dynamic param]) {
    final softMatrix = function(m);
    // Jacobian for each (mini)batch sample
    return List<Matrix>.generate(
        m.m,
        (index) =>
            Matrix.diag(diag: softMatrix.getColumn(index).flattenList()) -
            softMatrix.getColumn(index) * softMatrix.getColumn(index).T);
  };
  name = 'softmax';
}