cpu_version/cpu_version library

Holds all the subexports of the cpu_version library.

Classes

ActivationFunction<T>

The abstract base class (or interface) for all activation functions.

Adagrad

Implements the Adagrad optimization algorithm. Adapts the learning rate individually for each parameter based on the accumulated history of squared gradients.

Adam

Implements the Adam (Adaptive Moment Estimation) optimization algorithm. Computes individual adaptive learning rates from exponentially decaying averages of past gradients (first moment) and squared gradients (second moment), incorporating bias correction terms.

AdamW

Implements the AdamW optimization algorithm with decoupled weight decay. Modifies the standard Adam update by applying weight decay directly to the parameters rather than incorporating it into the gradient moments.

AMSGrad

Implements the AMSGrad optimization algorithm, a variant of Adam. Maintains the maximum of past squared gradient averages in _vHat to prevent learning rate scaling from exploding, ensuring non-increasing effective step sizes for improved convergence.

AveragePooling2DLayer

Applies a 2D average pooling operation over an input Matrix tensor. Downsamples the input representation by taking the average value over pooling windows defined by poolSize and stride.

BatchNorm1D

Applies Batch Normalization over a 1D Vector tensor. Normalizes the input features across the batch to have zero mean and unit variance, followed by a learnable affine transformation using gamma and beta. Tracks running statistics runningMean and runningVariance for evaluation during inference.

BatchNorm2D

Applies Batch Normalization over a 3D Tensor3D tensor across channels. Normalizes the channel dimensions across spatial elements to have zero mean and unit variance, followed by a learnable affine transformation using gamma and beta. Tracks running statistics runningMean and runningVariance for evaluation during inference.

Conv2DLayer

Applies a 2D convolution over an input Matrix tensor. Convolves the input with a set of learnable 2D filters (kernels) to produce a 3D feature map (Tensor3D), adds channel-wise biases, and optionally applies an activation function.

ConvLSTMLayer

Applies a Convolutional Long Short-Term Memory (ConvLSTM) recurrent step over a sequence of 2D matrices formatted as a Tensor3D. Replaces standard matrix multiplications in an LSTM with 2D convolutions (conv2d) using 'same' padding to preserve spatial dimensions across recurrent state transitions. Computes forget, input, cell, and output gates sequentially over the sequence length and returns the final hidden state Matrix.

DenseLayer

Represents a standard fully connected layer operating on a 1D Vector tensor. Performs a matrix-vector multiplication with weights, adds a biases vector, and optionally applies an activation function to the resulting vector.

DenseLayerMatrix

Represents a standard fully connected layer operating on a 2D Matrix tensor for batched inputs. Performs matrix multiplication of input with weights, broadcasts the biases vector across rows, and optionally applies an activation function to the output matrix.

DropoutLayer

Applies Dropout regularization over an input Vector tensor. During training, randomly zeroes out elements of the input tensor with probability rate and scales the remaining elements by 1.0 / (1.0 - rate) to maintain expected activation magnitude. During evaluation (isTraining = false), the input tensor is passed through unchanged.

DropoutLayerMatrix

Applies Dropout regularization over a 2D Matrix tensor for batched or spatial representations. During training, randomly zeroes out elements of the input matrix with probability rate and scales the remaining elements by 1.0 / (1.0 - rate) to maintain expected activation magnitude. During evaluation (isTraining = false), the input tensor is passed through unchanged.

DualLSTMLayer

A Multi-Timeline Long Short-Term Memory (MT-LSTM) layer operating over a 2D Matrix sequence. Maintains two separate tiers of LSTM cells running at different temporal granularities:

ELUMatrix

Applies the Exponential Linear Unit (ELU) activation function over a given input of type Tensor<Matrix>. In contrast to the standard ReLU function, this function has a small negative value for negative inputs, which can help prevent the dying ReLU problem and speed up learning by pushing the mean activation closer to zero.

ELUVector

Applies the Exponential Linear Unit (ELU) activation function over a given input of type Tensor<Vector>. In contrast to the standard ReLU function, this function has a small negative value for negative inputs, which can help prevent the dying ReLU problem and speed up learning by pushing the mean activation closer to zero.

EmbeddingLayer

Maps a 1D Vector tensor of integer indices (token IDs) to continuous embedding vectors, outputting a 2D Matrix tensor. Operates as an embedding lookup table of shape [vocabularySize, embeddingDimension] on the CPU with dedicated autograd backpropagation.

EmbeddingLayerMatrix

Maps a batched 2D Matrix tensor of integer indices [batchSize, sequenceLength] to dense continuous embeddings, outputting a 3D Tensor3D tensor. Operates as a batched embedding lookup table of shape [vocabularySize, embeddingDimension] on the CPU with dedicated autograd backpropagation.

FlattenLayer

Flattens a 2D Matrix tensor into a contiguous 1D Vector tensor. Preserves the underlying flat data elements while resetting dimensionality and attaching an autograd Node to propagate gradients backward.

GlobalAveragePooling1D

Applies a 1D global average pooling operation over an input Matrix sequence of shape [sequenceLength, numFeatures]. Computes the mean across all sequence timesteps for each feature dimension, reducing the 2D Matrix to a 1D Vector of size [numFeatures]. Registers an autograd Node to evenly distribute incoming gradients back across all sequence steps.

GlobalAveragePoolingLayer

Applies a global average pooling operation over an input Matrix tensor. Computes the mean along the spatial dimensions to reduce a 2D Matrix to a 1D Vector.

Layer<I, O>

Layer is built to be an additional abstraction that allows for complex mathematical bundling on the CPU. Operations like Dense Neural Networks are supported as a single Layer and can be used to build models reliably. Each Layer has the following individually assigned values:

LayerNormalization

Applies Layer Normalization over a 2D Matrix tensor across features. Normalizes the activations of each sample (row) across the feature dimension (columns) to have zero mean and unit variance, followed by a learnable affine transformation using gamma and beta. Constructs an autograd Node to backpropagate gradients to inputs and parameters.

LeakyReLUMatrix

Applies leaky ReLu over a given input of type Tensor<Matrix>. In contrast to the standard ReLU function, this function multiplies negative values with a small negative number alpha to prevent the dying gradient problem.

LeakyReLUVector

Applies leaky ReLu over a given input of type Tensor<Vector>. In contrast to the standard ReLU function, this function multiplies negative values with a small negative number alpha to prevent the dying gradient problem.

LSTMLayer

Applies a Long Short-Term Memory (LSTM) recurrent neural network pass over a sequential 2D Matrix tensor. Processes input sequences of shape [sequenceLength, featureSize] timestep-by-timestep using standard gating mechanisms (forget gate, input gate, candidate cell state, and output gate) to maintain long-term dependencies. Returns the final hidden state Vector after processing all sequence steps.

MaxPooling1DLayer

Applies a 1D max pooling operation over an input Vector tensor. Downsamples the 1D representation by selecting the maximum value within sliding windows defined by poolSize and stride. Tracks input indices of maximum values to route backpropagating gradients directly to those elements via an autograd Node.

MaxPooling2DLayer

Applies a 2D max pooling operation over an input Matrix tensor. Downsamples the spatial representation by extracting the maximum value within pooling windows defined by poolSize and stride. Tracks the flat 1D input indices of maximum values to route backpropagating gradients directly to those elements via an autograd Node.

MishMatrix

Applies the mish activation function (similar to swish) to each element inside of a given input of type Tensor<Matrix>. Mish mirrors Relu with a continuous curve. The calculation is handled via the mishMatrix math operation.

MishVector

Applies the mish activation function (similar to swish) to each element inside of a given input of type Tensor<Vector>. Mish mirrors Relu with a continuous curve. The calculation is handled via the mishVector math operation.

Momentum

Implements the Stochastic Gradient Descent with Momentum optimization algorithm. Accelerates gradient vectors in the relevant direction by accumulating past velocity updates, dampening oscillations across optimization steps.

MultiHeadAttention

Applies Multi-Head Attention over an input Matrix tensor. Projects the input sequence into multiple subspaces by running numHeads parallel SingleHeadAttention modules, concatenates the head outputs along the column dimension, and linearly projects the result back to dModel using wo.

MultiTierLSTMLayer

Applies a Multi-Tier Hierarchical Long Short-Term Memory recurrent pass over a 2D Matrix sequence. Extends multi-timeline recurrence to an arbitrary number of stacked tiers defined by tierClockCycles.

NAG

Implements Nesterov Accelerated Gradient (NAG) optimization. Incorporates momentum by computing the parameter update step using a look-ahead velocity term.

Optimizer

The abstract base class for all optimization algorithms. Tracks a collection of trainable parameters and manages optimization steps and gradient resets.

PositionalEncoding

Injects sinusoidal positional encodings into an input sequence Matrix tensor. Generates fixed sine and cosine frequency encodings up to maxLength across dModel feature dimensions to encode token positions, adding them directly to the input tensor.

ReLU

An activation function that applies the Rectified Linear Unit (ReLU) to a Vector.

ReLULayer

Applies the Rectified Linear Unit (ReLU) activation function element-wise over an input Vector tensor. Computes max(0, x) on each element of the input vector.

ReLULayerMatrix

Applies the Rectified Linear Unit (ReLU) activation function element-wise over a 2D Matrix tensor. Computes max(0, x) on each element across all rows and columns of the matrix.

ReLUMatrix

An activation function that applies the Rectified Linear Unit (ReLU) to a Matrix.

RMSprop

Implements the RMSprop optimization algorithm. Maintains a moving average of squared gradients to normalize gradient magnitudes, dividing the learning rate by the square root of recent gradient variances.

RNN

Applies a standard Elman Recurrent Neural Network (RNN) pass over a sequential 2D Matrix tensor. Processes input sequences of shape [sequenceLength, featureSize] sequentially across timesteps, updating the recurrent hidden state vector h via input-to-hidden projections (wXh), hidden-to-hidden transitions (wHh), and a bias vector (bh), followed by an activation function. Returns the final hidden state Vector after processing all sequence steps.

SGD

Implements the Stochastic Gradient Descent (SGD) optimization algorithm. Updates parameters in the opposite direction of the gradient scaled directly by learningRate.

Sigmoid

The Sigmoid activation function.

SigmoidMatrix

An activation function that applies the Sigmoid function to a Matrix.

SingleHeadAttention

Applies single-head scaled dot-product attention over an input sequence Matrix tensor. Projects input token embeddings into Query (wq), Key (wk), and Value (wv) representations, computes attention scores scaled by 1.0 / sqrt(dK), applies softmax normalization to extract attention weights, and computes the context matrix by weighting wv.

SNetwork

A sequential neural network architecture that linearly stacks and executes Layer instances. Manages sequential forward propagation, parameter aggregation across constituent layers, training loops via fit, weight serialization to JSON via save and load, and autograd graph visualization.

SoftmaxMatrix

Applies the softmax function over a given input of type Tensor<Matrix>. Creates a probability distribution over all values of each row summing up to 1. The calculation is handled via the softmaxMatrix math operation.

SoftmaxVector

Applies the softmax function over a given input of type Tensor<Vector>. Creates a probability distribution over all values summing up to 1. The calculation is handled via the softmaxVector math operation.

SwishMatrix

Applies the Sigmoid-weighted Linear Unit (SiLU / Swish) activation function to each element inside of a given input of type Tensor<Matrix>. Defined mathematically as f(x) = x * sigmoid(x), this modern activation function often outperforms standard ReLU in deeper hidden layers.

SwishVector

Applies the Sigmoid-weighted Linear Unit (SiLU / Swish) activation function to each element inside of a given input of type Tensor<Vector>. Defined mathematically as f(x) = x * sigmoid(x), this modern activation function often outperforms standard ReLU in deeper hidden layers.

TransformerEncoderBlock

Applies a standard Transformer Encoder Block over an input sequence Matrix tensor. Combines a multi-head self-attention module (mha) with a position-wise feed-forward network (ffn). Each sub-layer incorporates residual connections (addMatrix) followed by layer normalization (LayerNormalization).

Functions

flatMse(Tensor pred, Tensor target) → Tensor<Scalar>

Computes the Mean Squared Error (MSE) loss between flattened elements of pred and target tensors. Builds an autograd Node attached to the returned Scalar Tensor to propagate gradients back to pred.

main() → Future<void>

Demonstrates training, saving, loading, and evaluating an SNetwork model on the XOR problem.