core/nn/llama_vision library
Vision-conditioned Llama — from-scratch multi-modal wrapper.
Wires together three pieces:
- A ViTBackbone that produces per-patch features
(
[numPatches + 1, vitEmbedDim]) from a patchified image. - A VisionProjector that maps those features into Llama's
embedding space (
[numPatches + 1, llamaEmbedDim]). - A Llama decoder that consumes
[image_tokens; text_tokens]as a single causal sequence.
The image tokens sit before the text tokens in the sequence, so the causal mask lets every text token attend to all image tokens, but not vice versa. This matches the LLaVA / IDEFICS convention.
This is a "from-scratch demo" wiring:
- ViT weights are randomly initialised — this class does not pretend to load CLIP or any pretrained vision encoder.
- Llama weights can be either randomly initialised (fast smoke
demo) or loaded via
loadLlamaSafetensorsInto. - The projector is always initialised fresh and is what a training loop actually optimises (LLaVA-style stage-1 alignment).
See bin/train_llama_vision_demo.dart for the accompanying
tiny-caption training loop.