evaluationFunction property

String? evaluationFunction
final

Required. Python function. Expected user to define the following function, e.g.: def evaluate(instance: dictstr, Any) -> float: Please include this function signature in the code snippet. Instance is the evaluation instance, any fields populated in the instance are available to the function as instancefield_name.

Example: Example input:

instance= EvaluationInstance(
    response=EvaluationInstance.InstanceData(text="The answer is 4."),
    reference=EvaluationInstance.InstanceData(text="4")
)

Example converted input:

{
 'response': {'text': 'The answer is 4.'},
 'reference': {'text': '4'}
}

Example python function:

 def evaluate(instance: dict[str, Any]) -> float:
   if instance['response']['text'] == instance['reference']['text']:
     return 1.0
   return 0.0

CustomCodeExecutionSpec is also supported in Batch Evaluation (EvalDataset RPC) and Tuning Evaluation. Each line in the input jsonl file will be converted to dictstr, Any and passed to the evaluation function.

Implementation

final String? evaluationFunction;