evaluationFunction property
Python function.
Expected user to define the following function, e.g.: def
evaluate(instance: dict[str, Any]) -> float: Please include this
function signature in the code snippet. Instance is the evaluation
instance, any fields populated in the instance are available to the
function as instance[field_name]. Example: Example input: instance= EvaluationInstance( response=EvaluationInstance.InstanceData(text="The answer is 4."), reference=EvaluationInstance.InstanceData(text="4") )
Example converted input: { 'response': {'text': 'The answer is 4.'}, 'reference': {'text': '4'} } Example python function: def evaluate(instance: dict[str, Any]) -> float: if instance'response' == instance'reference': return 1.0 return 0.0 CustomCodeExecutionSpec is
also supported in Batch Evaluation (EvalDataset RPC) and Tuning
Evaluation. Each line in the input jsonl file will be converted to
dict[str, Any] and passed to the evaluation function.
Required.
Implementation
core.String? evaluationFunction;