getBenchmarkCompareWithHttpInfo method

Future<Response> getBenchmarkCompareWithHttpInfo(String a, String b, { String? benchmark, })

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p.

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p. Pairing is what prevents the subset artifact — comparing one model's easy subset against another's full run — so n_common, not either arm's own coverage, is the number to read this by. Both a and b are required. The benchmark defaults to gpqa_diamond.

Note: This method returns the HTTP Response.

Parameters:

• String a (required): A is the first model id. It is required.

• String b (required): B is the second model id. It is required.

• String benchmark: Benchmark is the catalog id to compare on, defaulting to gpqa_diamond.

Implementation

Future<Response> getBenchmarkCompareWithHttpInfo(String a, String b, { String? benchmark, }) async {
  // ignore: prefer_const_declarations
  final path = r'/v1/benchmark/compare';

  // ignore: prefer_final_locals
  Object? postBody;

  final queryParams = <QueryParam>[];
  final headerParams = <String, String>{};
  final formParams = <String, String>{};

  if (benchmark != null) {
    queryParams.addAll(_queryParams('', 'benchmark', benchmark));
  }
    queryParams.addAll(_queryParams('', 'a', a));
    queryParams.addAll(_queryParams('', 'b', b));

  const contentTypes = <String>[];


  return apiClient.invokeAPI(
    path,
    'GET',
    queryParams,
    postBody,
    headerParams,
    formParams,
    contentTypes.isEmpty ? null : contentTypes.first,
  );
}