core/nn/modalities/multi_modal_encoder library

Multimodal encoder — fuses per-modality sequences into a joint sequence context, suitable for feeding into a text decoder for tasks like captioning or spoken-Q&A.

Mid-fusion recipe (modalities interact through self-attention after being tokenized to a common jointEmbedDim):

  1. Each unimodal encoder runs independently and produces a sequence of length seq_i at width jointEmbedDim.
  2. Sequences are concat-ed along the row axis into a single [seqA + seqV + seqT, jointEmbedDim] block.
  3. A fusion TransformerEncoder self-attends across the merged block so features from different modalities can talk to each other.

All three unimodal encoders must produce features of the same jointEmbedDim — the constructor asserts this. Text is optional.

The returned tensor is [seqTotal, jointEmbedDim]. Wire it into a TransformerDecoder (via EncoderDecoder) if you want a generative head; feed it into another pooling + MLP layer for classification.