otelconnect

README

otelconnect

connectrpc.com/otelconnect adds support for OpenTelemetry tracing and metrics collection to Connect servers and clients.

For more on Connect, OpenTelemetry, and otelconnect, see the Connect announcement blog post and the observability documentation on connectrpc.com.

An example

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"

	"connectrpc.com/connect"
	"connectrpc.com/otelconnect"
	// Generated from your protobuf schema by protoc-gen-go and
	// protoc-gen-connect-go.
	pingv1 "connectrpc.com/otelconnect/internal/gen/observability/ping/v1"
	"connectrpc.com/otelconnect/internal/gen/observability/ping/v1/pingv1connect"
)

func main() {
	mux := http.NewServeMux()
	otelInterceptor, err := otelconnect.NewInterceptor()
	if err != nil {
		log.Fatal(err)
	}

	// otelconnect.NewInterceptor provides an interceptor that adds tracing and
	// metrics to both clients and handlers. By default, it uses OpenTelemetry's
	// global TracerProvider and MeterProvider, which you can configure by
	// following the OpenTelemetry documentation. If you'd prefer to avoid
	// globals, use otelconnect.WithTracerProvider and
	// otelconnect.WithMeterProvider.
	mux.Handle(pingv1connect.NewPingServiceHandler(
		&pingv1connect.UnimplementedPingServiceHandler{},
		connect.WithInterceptors(otelInterceptor),
	))

	http.ListenAndServe("localhost:8080", mux)
}

func makeRequest() {
	otelInterceptor, err := otelconnect.NewInterceptor()
	if err != nil {
		log.Fatal(err)
	}

	client := pingv1connect.NewPingServiceClient(
		http.DefaultClient,
		"http://localhost:8080",
		connect.WithInterceptors(otelInterceptor),
	)
	resp, err := client.Ping(
		context.Background(),
		connect.NewRequest(&pingv1.PingRequest{}),
	)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(resp)
}

Configuration for internal services

By default, instrumented servers are conservative and behave as though they're internet-facing. They don't trust any tracing information sent by the client, and will create new trace spans for each request. The new spans are linked to the remote span for reference (using OpenTelemetry's trace.Link), but tracing UIs will display the request as a new top-level transaction.

If your server is deployed as an internal service, configure otelconnect to trust the client's tracing information using otelconnect.WithTrustRemote. With this option, servers will create child spans for each request.

Reducing metrics and tracing cardinality

By default, the OpenTelemetry RPC conventions produce high-cardinality server-side metric and tracing output. In particular, servers tag all metrics and trace data with the server's IP address and the remote port number. To drop these attributes, use otelconnect.WithoutServerPeerAttributes. For more customizable attribute filtering, use otelconnect.WithFilter.

Status

	Unary	Streaming Client	Streaming Handler
Metrics	✅	✅	✅
Tracing	✅	✅	✅

Ecosystem

• connect-go: Service handlers and clients for Go
• connect-swift: Swift clients for idiomatic gRPC & Connect RPC
• connect-kotlin: Kotlin clients for idiomatic gRPC & Connect RPC
• connect-es: Type-safe APIs with Protobuf and TypeScript.
• Buf Studio: web UI for ad-hoc RPCs
• conformance: Connect, gRPC, and gRPC-Web interoperability tests

Support and Versioning

otelconnect supports:

• The two most recent major releases of Go.
• v1 of the go.opentelemetry.io/otel tracing and metrics SDK.

Legal

Offered under the Apache 2 license.

Documentation

Overview

Package otelconnect provides OpenTelemetry tracing and metrics for connectrpc.com/connect servers and clients. The specification followed was the OpenTelemetry specification with both the rpc metrics specification and rpc spans specification implemented.

Index

• Variables
• type AttributeFilter
• type Interceptor
  • func NewInterceptor(options ...Option) (*Interceptor, error)
  • func (i *Interceptor) WrapStreamingClient(next connect.StreamingClientFunc) connect.StreamingClientFunc
  • func (i *Interceptor) WrapStreamingHandler(next connect.StreamingHandlerFunc) connect.StreamingHandlerFunc
  • func (i *Interceptor) WrapUnary(next connect.UnaryFunc) connect.UnaryFunc
• type Option
  • func WithAttributeFilter(filter AttributeFilter) Option
  • func WithFilter(filter func(context.Context, connect.Spec) bool) Option
  • func WithMeterProvider(provider metric.MeterProvider) Option
  • func WithPropagateResponseHeader() Option
  • func WithPropagator(propagator propagation.TextMapPropagator) Option
  • func WithRPCSystem(system RPCSystem) Option
  • func WithTraceRequestHeader(keys ...string) Option
  • func WithTraceResponseHeader(keys ...string) Option
  • func WithTracerProvider(provider trace.TracerProvider) Option
  • func WithTrustRemote() Option
  • func WithoutMetrics() Option
  • func WithoutServerPeerAttributes() Option
  • func WithoutTraceEvents() Option
  • func WithoutTracing() Option
• type RPCSystem

Variables

var (
	// ConnectRPCSystem indicates that the semantic conventions for
	// ConnectRPC should be used.
	ConnectRPCSystem = connectRPCSystem{} //nolint:gochecknoglobals
	// GRPCSystem indicates that the semantic conventions for gRPC
	// should be used.
	GRPCSystem = gRPCSystem{} //nolint:gochecknoglobals
)

Types

type AttributeFilter

type AttributeFilter func(connect.Spec, attribute.KeyValue) bool

AttributeFilter is used to filter attributes out based on the connect.Spec and attribute.KeyValue. If the filter returns true the attribute will be kept else it will be removed. AttributeFilter must be safe to call concurrently.

type Interceptor

type Interceptor struct {
	// contains filtered or unexported fields
}

Interceptor implements connect.Interceptor that adds OpenTelemetry metrics and tracing to connect handlers and clients.

func NewInterceptor

func NewInterceptor(options ...Option) (*Interceptor, error)

NewInterceptor returns an interceptor that implements connect.Interceptor. It adds OpenTelemetry metrics and tracing to connect handlers and clients. Use options to configure the interceptor. Any invalid options will cause an error to be returned. The interceptor will use the default tracer and meter providers. To use a custom tracer or meter provider pass in the WithTracerProvider or WithMeterProvider options. To disable metrics or tracing pass in the WithoutMetrics or WithoutTracing options.

func (*Interceptor) WrapStreamingClient

func (i *Interceptor) WrapStreamingClient(next connect.StreamingClientFunc) connect.StreamingClientFunc

WrapStreamingClient implements otel tracing and metrics for streaming connect clients.

func (*Interceptor) WrapStreamingHandler

func (i *Interceptor) WrapStreamingHandler(next connect.StreamingHandlerFunc) connect.StreamingHandlerFunc

WrapStreamingHandler implements otel tracing and metrics for streaming connect handlers.

func (*Interceptor) WrapUnary

func (i *Interceptor) WrapUnary(next connect.UnaryFunc) connect.UnaryFunc

WrapUnary implements otel tracing and metrics for unary handlers.

type Option

type Option interface {
	// contains filtered or unexported methods
}

An Option configures the OpenTelemetry instrumentation.

func WithAttributeFilter

func WithAttributeFilter(filter AttributeFilter) Option

WithAttributeFilter sets the attribute filter for all metrics and trace attributes.

func WithFilter

func WithFilter(filter func(context.Context, connect.Spec) bool) Option

WithFilter configures the instrumentation to emit traces and metrics only when the filter function returns true. Filter functions must be safe to call concurrently.

func WithMeterProvider

func WithMeterProvider(provider metric.MeterProvider) Option

WithMeterProvider configures the instrumentation to use the supplied metric.MeterProvider when extracting and injecting trace context. By default, the instrumentation uses otel.GetMeterProvider.

func WithPropagateResponseHeader

func WithPropagateResponseHeader() Option

WithPropagateResponseHeader enables injecting the traceparent header into response headers for server-side interceptors. This allows clients to correlate their requests with server-side traces.

func WithPropagator

func WithPropagator(propagator propagation.TextMapPropagator) Option

WithPropagator configures the instrumentation to use the supplied propagator when extracting and injecting trace context. By default, the instrumentation uses otel.GetTextMapPropagator.

func WithRPCSystem

func WithRPCSystem(system RPCSystem) Option

WithRPCSystem forces the use of the semantic conventions for the given RPC system. By default, the conventions used vary based on the actual protocol of a request: so requests that a client sends or a server receives that use the gRPC or gRPC-Web protocols use the gRPC semantic conventions; requests that use ConnectRPC use the ConnectRPC semantic conventions.

In a system where a server handles requests for the same service but from clients that use multiple protocols, this causes the telemetry data to be partitioned by the RPC system conventions. But it is often desirable to instead emit uniform telemetry, to allow optics and aggregation across RPC systems.

So this option can be used to force uniform metrics and spans, using the given RPC system conventions, regardless of the actual protocol.

func WithTraceRequestHeader

func WithTraceRequestHeader(keys ...string) Option

WithTraceRequestHeader enables header attributes for the request header keys provided. Attributes will be added as Trace attributes only.

func WithTraceResponseHeader

func WithTraceResponseHeader(keys ...string) Option

WithTraceResponseHeader enables header attributes for the response header keys provided. Attributes will be added as Trace attributes only.

func WithTracerProvider

func WithTracerProvider(provider trace.TracerProvider) Option

WithTracerProvider configures the instrumentation to use the supplied provider when creating a tracer. By default, the instrumentation uses otel.GetTracerProvider.

func WithTrustRemote

func WithTrustRemote() Option

WithTrustRemote sets the Interceptor to trust remote spans. By default, all incoming server spans are untrusted and will be linked with a trace.Link and will not be a child span. By default, all client spans are trusted and no change occurs when WithTrustRemote is used.

func WithoutMetrics

func WithoutMetrics() Option

WithoutMetrics disables metrics.

func WithoutServerPeerAttributes

func WithoutServerPeerAttributes() Option

WithoutServerPeerAttributes removes net.peer.port and net.peer.name attributes from server trace and span attributes. The default behavior follows the OpenTelemetry semantic conventions for RPC, but produces very high-cardinality data; this option significantly reduces cardinality in most environments.

func WithoutTraceEvents

func WithoutTraceEvents() Option

WithoutTraceEvents disables trace events for both unary and streaming interceptors. This reduces the quantity of data sent to your tracing system by omitting per-message information like message size.

func WithoutTracing

func WithoutTracing() Option

WithoutTracing disables tracing.

type RPCSystem

type RPCSystem interface {
	// contains filtered or unexported methods
}

RPCSystem represents an RPC system, like ConnectRPC or gRPC. Different systems have different semantic conventions for how metrics and spans are defined.

https://opentelemetry.io/docs/specs/semconv/rpc/

Valid values currently are ConnectRPCSystem and GRPCSystem.