OpenTelemetry in Go without the ceremony

OpenTelemetry is the observability standard now. Not a contender — the standard. Every major cloud provider, every monitoring vendor, and the Go standard library are all converging on OTLP as the wire format. If you're starting a new Go backend in 2026, instrumenting it with OpenTelemetry from day one costs almost nothing and pays off the first time you need to debug a production issue you can't reproduce locally.

Why OpenTelemetry won

Before OpenTelemetry, instrumenting a service meant picking a vendor SDK and being locked in. Switching from Datadog to Honeycomb meant re-instrumenting the codebase. OpenTelemetry separated the instrumentation (how you record telemetry) from the backend (where you send it). You instrument once, configure the exporter, and point it wherever you want.

The Go SDK is stable and actively maintained. The auto-instrumentation libraries cover most of the common cases — HTTP servers, database clients, gRPC — so you get value before writing a single custom span.

The three signals

• Traces— a request's journey through your system. A trace is a tree of spans; each span is a unit of work with a start time, duration, and attributes. Essential for understanding latency and request flow across services.
• Metrics — numeric measurements over time. Request rates, error rates, queue depths, DB connection pool usage. Alerting lives here.
• Logs— structured log lines correlated to trace IDs. OTel doesn't replace your logger, it correlates it.

Setup in Go

Initialize the tracer and meter providers in main.go, before the server starts. Keep the setup in a dedicated package.

func initOtel(ctx context.Context, svcName, endpoint string) (shutdown func(), err error) {
  res := resource.NewWithAttributes(
    semconv.SchemaURL,
    semconv.ServiceName(svcName),
    semconv.ServiceVersion("1.0.0"),
  )
  // OTLP gRPC exporter — works with Grafana, Honeycomb, etc.
  exp, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint(endpoint),
    otlptracegrpc.WithInsecure(),
  )
  if err != nil { return nil, err }

  tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exp),
    sdktrace.WithResource(res),
    sdktrace.WithSampler(sdktrace.AlwaysSample()),
  )
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.TraceContext{})
  return func() { tp.Shutdown(ctx) }, nil
}

Adding traces

Wrap your HTTP middleware with the OTel HTTP handler for automatic span creation on every request:

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

// Wraps the entire router — every request gets a root span.
handler := otelhttp.NewHandler(router, "api")

For database calls, add a child span manually. Keep spans narrow — one span per SQL query, not one span for the entire request:

tracer := otel.Tracer("invoice-store")

func (s *Store) GetInvoice(ctx context.Context, id string) (*Invoice, error) {
  ctx, span := tracer.Start(ctx, "GetInvoice")
  defer span.End()

  span.SetAttributes(attribute.String("invoice.id", id))
  inv, err := s.queries.GetInvoice(ctx, id)
  if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
  }
  return inv, err
}

Metrics

Use the OTel metrics SDK for business-level signals. The HTTP middleware above auto-generates request count and latency histograms. Add custom counters for domain events:

meter := otel.Meter("invoice-service")

invoicesCreated, _ := meter.Int64Counter("invoices.created",
  metric.WithDescription("Total invoices created"),
)

// In your service, after creating an invoice:
invoicesCreated.Add(ctx, 1, metric.WithAttributes(
  attribute.String("currency", inv.Currency),
))

Where to send the data

• Local dev: grafana/otel-lgtm Docker image — Grafana + Tempo + Loki + Mimir, pre-wired. One docker-compose line, full observability stack.
• Small production: Grafana Cloud free tier. 14-day trace retention, 10k series free. Enough to get started.
• If you want the best DX: Honeycomb. Expensive, but the query interface for traces is genuinely the best in the industry.
• Self-hosted at scale: OTel Collector → Tempo (traces) + Prometheus (metrics) + Loki (logs). The standard open-source stack.

Observability is one of the things that's cheap to add on day one and expensive to retrofit into a running system. The deploy pipeline post covers the rest of the production readiness checklist.