core/nn/gpt2_hf_loader library

Loads HuggingFace gpt2 / gpt2-medium / gpt2-large / gpt2-xl weights into a GPT module.

The expected on-disk layout is a HuggingFace safetensors file produced by e.g.:

from transformers import GPT2LMHeadModel
m = GPT2LMHeadModel.from_pretrained("gpt2")
m.save_pretrained("gpt2", safe_serialization=True)
# -> gpt2/model.safetensors

or downloaded straight from the HF Hub (model.safetensors).

Weight conventions (HF → dart_pytorch):

  • wte.weight [V, D] → tokenEmb.weight
  • wpe.weight [maxCtx, D] → posEmb.table.weight
  • h.{i}.ln_1.weight/bias [D] → block.ln1.gamma/beta
  • h.{i}.attn.c_attn.weight [D, 3D] → block QKV per-head weights
  • h.{i}.attn.c_attn.bias [3D] → block QKV per-head biases
  • h.{i}.attn.c_proj.weight [D, D] → block mha.wo.weight
  • h.{i}.attn.c_proj.bias [D] → block mha.wo.bias
  • h.{i}.ln_2.weight/bias [D] → block.ln2.gamma/beta
  • h.{i}.mlp.c_fc.weight [D, 4D] → block.ffn1.weight
  • h.{i}.mlp.c_fc.bias [4D] → block.ffn1.bias
  • h.{i}.mlp.c_proj.weight [4D, D] → block.ffn2.weight
  • h.{i}.mlp.c_proj.bias [D] → block.ffn2.bias
  • ln_f.weight/bias [D] → encoder.finalNorm.gamma/beta

GPT-2 uses Conv1D layers (weight shape [in, out]), so every _.weight matrix is transposed on load.

The lm_head.weight is tied to wte.weight in HF; when the target GPT has tieWeights: true (the default) it is loaded implicitly via wte. When untied, the loader falls back to wte for the head as well (HF does not save a separate head).

GPT-2 uses GELU (tanh approx) and biased attention projections; callers must build the target GPT with attnBias: true, activation: Activation.geluTanh or the outputs will diverge from the reference model.

Classes

GPT2HFLoader
GPT2LoadReport
Diagnostic summary of a single GPT2HFLoader.loadFile / loadMap call. unusedKeys typically contains extra HF conveniences like attn.bias (the causal mask buffer) or masked_bias; these are safe to ignore because we build the causal mask on the fly.