core/nn/clip_hf_loader library

Loads HuggingFace CLIP vision weights (CLIPVisionModel / CLIPModel safetensors) into a CLIPVisionModel.

Supported checkpoints (fp32 weight sizes; fp16 safetensors are promoted to fp32 by SafeTensors):

  • openai/clip-vit-base-patch32 — 12 layers, 768 embed, 12 heads, ffn 3072, image 224, patch 32 → 49 patches. ~350 MB fp32 (~150 MB fp16 on disk).
  • openai/clip-vit-base-patch16 — same config as B/32 but patch 16 → 196 patches. Same weight count.
  • openai/clip-vit-large-patch14 — 24 layers, 1024 embed, 16 heads, ffn 4096, image 224, patch 14 → 256 patches. ~1.2 GB fp32.

HF key conventions handled here (the top-level namespace is vision_model. for CLIPVisionModel safetensors, or vision_model. under the joint CLIPModel bundle; the loader auto-detects the prefix by probing for both):

  • vision_model.embeddings.patch_embedding.weight — [D, C, P, P] Conv2d weight, permuted to [D, P*P*C] (channels-last per pixel) to match our patchify layout, then assigned to the Linear projector.
  • vision_model.embeddings.class_embedding — [D]
  • vision_model.embeddings.position_embedding.weight — [numPatches + 1, D]
  • vision_model.pre_layrnorm.{weight,bias} — [D] each (HF does misspell it — the typo is in transformers).
  • vision_model.encoder.layers.{i}.self_attn.{q,k,v}_proj.{weight,bias} — [D, D] / [D]. Sliced row-wise into per-head [hd, D] / [hd].
  • vision_model.encoder.layers.{i}.self_attn.out_proj.{weight,bias} — [D, D] / [D]. Directly copied to wo.
  • vision_model.encoder.layers.{i}.layer_norm{1,2}.{weight,bias} — [D] each; mapped to ln1 (pre-attn) and ln2 (pre-FFN).
  • vision_model.encoder.layers.{i}.mlp.fc1.{weight,bias} — [F, D] / [F].
  • vision_model.encoder.layers.{i}.mlp.fc2.{weight,bias} — [D, F] / [D].
  • vision_model.post_layernorm.{weight,bias} — [D] each.

Ignored (safe): every key that doesn't start with vision_model. (text tower, logit_scale, visual_projection, etc.). They are returned in ClipLoadReport.unusedKeys so callers can log them.