resizeVocab method

void resizeVocab(
  1. int newVocabSize
)

Resize vocabulary to newVocabSize, preserving pretrained weights.

Call this after loading pretrained weights and adding special tokens so that embedTokens and lmHead can handle the expanded vocabulary.

Implementation

void resizeVocab(int newVocabSize) {
  embedTokens.resize(newVocabSize);
  // Rebuild lmHead with new output dimension, copying existing rows
  final oldLmHead = lmHead;
  lmHead = Linear(embedDim, newVocabSize, useBias: false);
  final copyRows = newVocabSize < oldLmHead.outFeatures
      ? newVocabSize
      : oldLmHead.outFeatures;
  final newData = Float32List(newVocabSize * embedDim);
  for (int i = 0; i < copyRows * embedDim; i++) {
    newData[i] = oldLmHead.weight.data[i];
  }
  lmHead.weight = Tensor(newData, [newVocabSize, embedDim]);
}