getBenchmarkCompare method

Future<Pairing?> getBenchmarkCompare(String a, String b, { String? benchmark, })

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p.

Is the ONLY valid arm-vs-arm test: it pairs the two models on the items BOTH completed, and answers rescue and damage counts with an exact-McNemar p. Pairing is what prevents the subset artifact — comparing one model's easy subset against another's full run — so n_common, not either arm's own coverage, is the number to read this by. Both a and b are required. The benchmark defaults to gpqa_diamond.

Parameters:

• String a (required): A is the first model id. It is required.

• String b (required): B is the second model id. It is required.

• String benchmark: Benchmark is the catalog id to compare on, defaulting to gpqa_diamond.

Implementation

Future<Pairing?> getBenchmarkCompare(String a, String b, { String? benchmark, }) async {
  final response = await getBenchmarkCompareWithHttpInfo(a, b,  benchmark: benchmark, );
  if (response.statusCode >= HttpStatus.badRequest) {
    throw ApiException(response.statusCode, await _decodeBodyBytes(response));
  }
  // When a remote server returns no body with a status of 204, we shall not decode it.
  // At the time of writing this, `dart:convert` will throw an "Unexpected end of input"
  // FormatException when trying to decode an empty string.
  if (response.body.isNotEmpty && response.statusCode != HttpStatus.noContent) {
    return await apiClient.deserializeAsync(await _decodeBodyBytes(response), 'Pairing',) as Pairing;

  }
  return null;
}