rpc_dart_opentelemetry 0.5.0

OpenTelemetry tracing, metrics, and logging integration for rpc_dart.

rpc_dart_opentelemetry

OpenTelemetry tracing and call metrics for rpc_dart, built on the opentelemetry package. It also mirrors LogController spans and events into OTel. Logs reach the backend as spans and span events; the OTel logs API is not used.

• OtelRpcInterceptor — server-side interceptor: one span per call, W3C traceparent extracted from the incoming metadata.
• OtelRpcClientInterceptor — caller-side counterpart, injects traceparent so the trace continues across the RPC boundary.
• RpcOtelPropagator — W3C trace-context extract/inject over RpcContext headers.
• RpcOtelMetrics — call counters and duration instruments.
• LogControllerOtelOutput — a LogOutput that turns the built-in logger's spans and events into OTel spans and span events.

Install

dependencies:
  rpc_dart_opentelemetry: ^0.4.0
  opentelemetry: ^0.18.0

Setup

1. Bootstrap the OTel SDK

import 'package:opentelemetry/api.dart';
import 'package:opentelemetry/sdk.dart';

TracerProviderBase bootstrapTracing() {
  // ConsoleExporter is for development; use CollectorExporter in production.
  final provider = TracerProviderBase(
    processors: [SimpleSpanProcessor(ConsoleExporter())],
  );
  registerGlobalTracerProvider(provider);
  return provider;
}

CollectorExporter ships in package:opentelemetry/sdk.dart. It sends OTLP over HTTP with protobuf to the exact URI you give it, so pass the collector's HTTP port and path, not the gRPC port 4317:

TracerProviderBase otlpTracing() => TracerProviderBase(
  processors: [
    BatchSpanProcessor(
      CollectorExporter(Uri.parse('http://localhost:4318/v1/traces')),
    ),
  ],
);

Call provider.shutdown() before the process exits so buffered spans are flushed.

2. Attach the interceptors

import 'package:opentelemetry/api.dart';
import 'package:rpc_dart/rpc_dart.dart';
import 'package:rpc_dart_opentelemetry/rpc_dart_opentelemetry.dart';

void instrument(RpcResponderEndpoint server, RpcCallerEndpoint client) {
  final tracer = globalTracerProvider.getTracer('my-service', version: '1.0.0');

  server.addInterceptor(OtelRpcInterceptor(tracer: tracer));
  client.addInterceptor(OtelRpcClientInterceptor(tracer: tracer));
}

Every call through either endpoint now produces a span.

Each interceptor takes only the calls of its side: the server the calls its endpoint serves, the client the calls it makes (RpcCallInfo.direction). So an RpcPeerEndpoint, which makes and serves calls through one interceptor list, takes both, and each call gets one span of the right kind:

peer
  ..addInterceptor(OtelRpcInterceptor(tracer: tracer))
  ..addInterceptor(OtelRpcClientInterceptor(tracer: tracer));

where: narrows the calls either one traces; a call it turns down passes through untraced. A server stream that lives as long as a session would be one span for all of it:

bool notFeed(RpcCallInfo call) => call.methodName != 'feed';
OtelRpcInterceptor(tracer: tracer, where: notFeed);

Spans

Each span is named <Service>/<Method>, with kind server (from OtelRpcInterceptor) or client (from OtelRpcClientInterceptor).

Attribute	Value
rpc.system	rpc_dart
rpc.service	service name
rpc.method	method name
rpc.call_type	unary, server_stream, client_stream, bidirectional_stream
rpc.trace_id	RpcContext.traceId, when present (server only)
rpc.grpc.status_code	numeric gRPC status, 0 = OK (int)
rpc.grpc.status	status name: OK, UNAVAILABLE, DEADLINE_EXCEEDED, ...
rpc.stream.messages	response messages emitted (server and bidirectional streams)

On success the span status is ok. On error the exception is recorded and the status is error; the code comes from RpcStatusException.statusCode. Any other error is INTERNAL (13) on the server, which is what its caller receives, and UNKNOWN (2) on the client.

A streaming span ends when the response stream completes or its subscriber cancels. A cancelled call, a stream its subscriber cancelled or a call that threw RpcCancelledException, ends with code CANCELLED (1). An error event in the middle of a stream does not end it: each error is recorded on the span, and the last one sets the final status. rpc.stream.messages counts the messages of a streamed response.

Both interceptors are RpcCallObserverInterceptors from rpc_dart.

Accessing the span in a handler

OtelRpcInterceptor stores the server span in the call's RpcContext under OtelRpcKeys.span:

Future<RpcString> _hello(RpcString request, {RpcContext? context}) async {
  final span = context?.getValue<Span>(OtelRpcKeys.span);
  span?.setAttribute(Attribute.fromString('greeting.name', request.value));
  return RpcString('Hello, ${request.value}');
}

The handler also runs with the server span as the ambient OTel context (Context.current), so spans started inside it nest under the RPC span.

Propagation

• Server: extraction is automatic. OtelRpcInterceptor calls RpcOtelPropagator.extract on every incoming call and parents its span on the caller's.
• Client: OtelRpcClientInterceptor injects traceparent / tracestate into every outgoing call.

Without the client interceptor, inject by hand. inject uses Context.current unless you pass context:, and returns the context unchanged when there is no active span. GreeterCaller is any caller contract whose method forwards context: to callUnary:

Future<RpcString> tracedHello(GreeterCaller greeter, Span parent) {
  final context = RpcOtelPropagator.inject(
    RpcContext.empty(),
    context: contextWithSpan(Context.current, parent),
  );
  return greeter.hello(const RpcString('Ada'), context: context);
}

Metrics

Pass RpcOtelMetrics to either interceptor. One instance can be shared by both sides; each side writes to its own instruments.

// Meter is part of the opentelemetry package's experimental API.
// ignore: implementation_imports
import 'package:opentelemetry/src/experimental_api.dart' show Meter;

void instrumentWithMetrics(
  RpcResponderEndpoint server,
  RpcCallerEndpoint client,
  Tracer tracer,
  Meter meter,
) {
  final metrics = RpcOtelMetrics(meter: meter);
  server.addInterceptor(OtelRpcInterceptor(tracer: tracer, metrics: metrics));
  client.addInterceptor(
    OtelRpcClientInterceptor(tracer: tracer, metrics: metrics),
  );
}

Instrument	Type	Unit
rpc.server.requests	Counter	{call}
rpc.server.duration	Counter (running sum)	ms
rpc.client.requests	Counter	{call}
rpc.client.duration	Counter (running sum)	ms

Every measurement carries rpc.system, rpc.service, rpc.method, rpc.grpc.status_code (int) and rpc.grpc.status (name). Names follow the OpenTelemetry RPC semantic conventions.

Duration is a sum counter, not a histogram: the opentelemetry Meter offers only createCounter. So the average is available, percentiles are not.

The opentelemetry 0.18 SDK's Counter.add records nothing, so a Meter from its MeterProvider exports no data. Supply a Meter implementation backed by your metrics pipeline.

A failure inside the meter never fails the call; it is recorded as a rpc.metrics.record_failed event on the span.

Bridging LogController into OpenTelemetry

LogControllerOtelOutput is a LogOutput that mirrors the built-in logger's spans (LogScope.startSpan, LogScope.withSpan) and events into OTel:

LogController buildLogger(Tracer tracer) {
  return LogController(
    outputs: [
      ConsoleOutput(),
      // Inside a handler Context.current is the RPC span, so log spans
      // started there become its children.
      LogControllerOtelOutput(tracer: tracer),
    ],
  );
}

LogController record	OpenTelemetry
LogSpanStart	tracer.startSpan(name) with the record's start time; child of the open parent span when parentSpanId is set
LogEvent with a known spanId	span.addEvent(message, attributes)
LogEvent without one	zero-duration span log.<scope>
LogSpan (end)	data as attributes, ok/error status, span.end(endTime)

Nested log spans become child OTel spans. A log span or standalone event with no open parent is parented on rootContextProvider(), which defaults to Context.current. So a withSpan('db.query', ...) inside a handler becomes a child of the server span from OtelRpcInterceptor, giving one connected trace from the caller down to the handler's own work. Pass rootContextProvider: only to choose a different root.

LogControllerOtelOutput does not use the OTel logs API. A log line inside an open log span becomes a span event; one outside any log span becomes its own zero-duration span. Keep the controller's minLevel at info or above, or every debug line becomes a span.

One trace across processes. A log span names its place in its trace as a W3C traceparent (LogSpanHandle.traceparent); a span another process opens with it (LogScope.startSpan(name, parent: traceparent)) is its child. A parent this output never opened, named by W3C ids, becomes a remote span context. For the remote child to find its parent in the backend, the parent's OTel span must carry the log span's ids: give the tracer provider a LogSpanIds and pass the same one as ids:.

final ids = LogSpanIds();
final provider = TracerProviderBase(processors: processors, idGenerator: ids);
final output = LogControllerOtelOutput(
  tracer: provider.getTracer('svc'),
  ids: ids,
);

Every span the output opens then has its log span's span id, and a root its trace id; other spans (an interceptor's, the app's own) keep random ids.

Inside an RPC handler a log span nests under the server's span in OTel; give the controller ambientParent: otelTraceparent so its traceparent names that trace too. A log span's links (startSpan(links:), the spans of the messages it answers) become SpanLinks.

Spans whose end record never arrives are bounded: past maxOpenSpans (default 1024) the oldest are ended, and a sweep every sweepInterval (default 30 s) ends any open longer than spanTtl (default 5 min). dispose() ends whatever is still open.