cpu_version/layertypes/layer library

Holds and exports all the different layer types that can be used for CPU model building.

Classes

AveragePooling2DLayer

Applies a 2D average pooling operation over an input Matrix tensor. Downsamples the input representation by taking the average value over pooling windows defined by poolSize and stride.

BatchNorm1D

Applies Batch Normalization over a 1D Vector tensor. Normalizes the input features across the batch to have zero mean and unit variance, followed by a learnable affine transformation using gamma and beta. Tracks running statistics runningMean and runningVariance for evaluation during inference.

BatchNorm2D

Applies Batch Normalization over a 3D Tensor3D tensor across channels. Normalizes the channel dimensions across spatial elements to have zero mean and unit variance, followed by a learnable affine transformation using gamma and beta. Tracks running statistics runningMean and runningVariance for evaluation during inference.

Conv2DLayer

Applies a 2D convolution over an input Matrix tensor. Convolves the input with a set of learnable 2D filters (kernels) to produce a 3D feature map (Tensor3D), adds channel-wise biases, and optionally applies an activation function.

ConvLSTMLayer

Applies a Convolutional Long Short-Term Memory (ConvLSTM) recurrent step over a sequence of 2D matrices formatted as a Tensor3D. Replaces standard matrix multiplications in an LSTM with 2D convolutions (conv2d) using 'same' padding to preserve spatial dimensions across recurrent state transitions. Computes forget, input, cell, and output gates sequentially over the sequence length and returns the final hidden state Matrix.

DenseLayer

Represents a standard fully connected layer operating on a 1D Vector tensor. Performs a matrix-vector multiplication with weights, adds a biases vector, and optionally applies an activation function to the resulting vector.

DenseLayerMatrix

Represents a standard fully connected layer operating on a 2D Matrix tensor for batched inputs. Performs matrix multiplication of input with weights, broadcasts the biases vector across rows, and optionally applies an activation function to the output matrix.

DropoutLayer

Applies Dropout regularization over an input Vector tensor. During training, randomly zeroes out elements of the input tensor with probability rate and scales the remaining elements by 1.0 / (1.0 - rate) to maintain expected activation magnitude. During evaluation (isTraining = false), the input tensor is passed through unchanged.

DropoutLayerMatrix

Applies Dropout regularization over a 2D Matrix tensor for batched or spatial representations. During training, randomly zeroes out elements of the input matrix with probability rate and scales the remaining elements by 1.0 / (1.0 - rate) to maintain expected activation magnitude. During evaluation (isTraining = false), the input tensor is passed through unchanged.

DualLSTMLayer

A Multi-Timeline Long Short-Term Memory (MT-LSTM) layer operating over a 2D Matrix sequence. Maintains two separate tiers of LSTM cells running at different temporal granularities:

EmbeddingLayer

Maps a 1D Vector tensor of integer indices (token IDs) to continuous embedding vectors, outputting a 2D Matrix tensor. Operates as an embedding lookup table of shape [vocabularySize, embeddingDimension] on the CPU with dedicated autograd backpropagation.

EmbeddingLayerMatrix

Maps a batched 2D Matrix tensor of integer indices [batchSize, sequenceLength] to dense continuous embeddings, outputting a 3D Tensor3D tensor. Operates as a batched embedding lookup table of shape [vocabularySize, embeddingDimension] on the CPU with dedicated autograd backpropagation.

FlattenLayer

Flattens a 2D Matrix tensor into a contiguous 1D Vector tensor. Preserves the underlying flat data elements while resetting dimensionality and attaching an autograd Node to propagate gradients backward.

GlobalAveragePooling1D

Applies a 1D global average pooling operation over an input Matrix sequence of shape [sequenceLength, numFeatures]. Computes the mean across all sequence timesteps for each feature dimension, reducing the 2D Matrix to a 1D Vector of size [numFeatures]. Registers an autograd Node to evenly distribute incoming gradients back across all sequence steps.

GlobalAveragePoolingLayer

Applies a global average pooling operation over an input Matrix tensor. Computes the mean along the spatial dimensions to reduce a 2D Matrix to a 1D Vector.

Layer<I, O>

Layer is built to be an additional abstraction that allows for complex mathematical bundling on the CPU. Operations like Dense Neural Networks are supported as a single Layer and can be used to build models reliably. Each Layer has the following individually assigned values:

LayerNormalization

Applies Layer Normalization over a 2D Matrix tensor across features. Normalizes the activations of each sample (row) across the feature dimension (columns) to have zero mean and unit variance, followed by a learnable affine transformation using gamma and beta. Constructs an autograd Node to backpropagate gradients to inputs and parameters.

LSTMLayer

Applies a Long Short-Term Memory (LSTM) recurrent neural network pass over a sequential 2D Matrix tensor. Processes input sequences of shape [sequenceLength, featureSize] timestep-by-timestep using standard gating mechanisms (forget gate, input gate, candidate cell state, and output gate) to maintain long-term dependencies. Returns the final hidden state Vector after processing all sequence steps.

MaxPooling1DLayer

Applies a 1D max pooling operation over an input Vector tensor. Downsamples the 1D representation by selecting the maximum value within sliding windows defined by poolSize and stride. Tracks input indices of maximum values to route backpropagating gradients directly to those elements via an autograd Node.

MaxPooling2DLayer

Applies a 2D max pooling operation over an input Matrix tensor. Downsamples the spatial representation by extracting the maximum value within pooling windows defined by poolSize and stride. Tracks the flat 1D input indices of maximum values to route backpropagating gradients directly to those elements via an autograd Node.

MultiHeadAttention

Applies Multi-Head Attention over an input Matrix tensor. Projects the input sequence into multiple subspaces by running numHeads parallel SingleHeadAttention modules, concatenates the head outputs along the column dimension, and linearly projects the result back to dModel using wo.

MultiTierLSTMLayer

Applies a Multi-Tier Hierarchical Long Short-Term Memory recurrent pass over a 2D Matrix sequence. Extends multi-timeline recurrence to an arbitrary number of stacked tiers defined by tierClockCycles.

PositionalEncoding

Injects sinusoidal positional encodings into an input sequence Matrix tensor. Generates fixed sine and cosine frequency encodings up to maxLength across dModel feature dimensions to encode token positions, adding them directly to the input tensor.

ReLULayer

Applies the Rectified Linear Unit (ReLU) activation function element-wise over an input Vector tensor. Computes max(0, x) on each element of the input vector.

ReLULayerMatrix

Applies the Rectified Linear Unit (ReLU) activation function element-wise over a 2D Matrix tensor. Computes max(0, x) on each element across all rows and columns of the matrix.

RNN

Applies a standard Elman Recurrent Neural Network (RNN) pass over a sequential 2D Matrix tensor. Processes input sequences of shape [sequenceLength, featureSize] sequentially across timesteps, updating the recurrent hidden state vector h via input-to-hidden projections (wXh), hidden-to-hidden transitions (wHh), and a bias vector (bh), followed by an activation function. Returns the final hidden state Vector after processing all sequence steps.

SingleHeadAttention

Applies single-head scaled dot-product attention over an input sequence Matrix tensor. Projects input token embeddings into Query (wq), Key (wk), and Value (wv) representations, computes attention scores scaled by 1.0 / sqrt(dK), applies softmax normalization to extract attention weights, and computes the context matrix by weighting wv.

TransformerEncoderBlock

Applies a standard Transformer Encoder Block over an input sequence Matrix tensor. Combines a multi-head self-attention module (mha) with a position-wise feed-forward network (ffn). Each sub-layer incorporates residual connections (addMatrix) followed by layer normalization (LayerNormalization).

Functions

flatMse(Tensor pred, Tensor target) → Tensor<Scalar>

Computes the Mean Squared Error (MSE) loss between flattened elements of pred and target tensors. Builds an autograd Node attached to the returned Scalar Tensor to propagate gradients back to pred.