LlmInferenceOptions.gpu constructor

LlmInferenceOptions.gpu({
  1. required String modelPath,
  2. required int sequenceBatchSize,
  3. required int maxTokens,
  4. required double temperature,
  5. required int topK,
  6. int decodeStepsPerSync = 3,
  7. int? randomSeed,
})

{@macro LlmInferenceOptions}

Constructor for inference models using the GPU.

Implementation

factory LlmInferenceOptions.gpu({
  required String modelPath,
  required int sequenceBatchSize,
  required int maxTokens,
  required double temperature,
  required int topK,
  int decodeStepsPerSync = 3,
  int? randomSeed,
}) =>
    throw UnimplementedError();