Telemetry

Configure distributed tracing, Prometheus metrics, and structured logging for agentgateway.

Agentgateway emits telemetry through three channels:

• Distributed traces via OTLP to any OpenTelemetry-compatible backend (Jaeger, Grafana Tempo, etc.)
• Prometheus metrics scraped from a configurable stats address
• Structured logs written to stdout with configurable level and format

Distributed tracing

Tracing is configured under config.tracing (static config file) or frontendPolicies.tracing (dynamic config). Send traces to any OTLP endpoint:

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: true

Or with the frontendPolicies form used in the telemetry example:

frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - mcp:
          targets:
          - name: everything
            stdio:
              cmd: npx
              args: ["@modelcontextprotocol/server-everything"]

Tracing fields

Field	Description
otlpEndpoint	Full OTLP endpoint URL, e.g. http://localhost:4317
host	OTLP host in host:port format (used in frontendPolicies)
otlpProtocol	OTLP transport protocol (grpc or http/protobuf)
path	OTLP path. Defaults to /v1/traces
randomSampling	CEL expression or boolean. Initiates a new trace when no incoming trace context is present. Defaults to false
clientSampling	CEL expression or boolean. Whether to continue a trace from an incoming traceparent header. Defaults to true
fields.add	Additional attributes to add to every span
fields.remove	Span attributes to strip before exporting

Sampling

randomSampling controls whether agentgateway starts new traces for requests that arrive without a trace context. It accepts:

• true / false — sample all or none
• A float between 0.0 and 1.0 — percentage of requests to sample
• A CEL expression that evaluates to a float or boolean

clientSampling controls whether to propagate traces that arrive with an existing traceparent header. It defaults to true (always propagate).

config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: 0.1   # Sample 10% of new requests
    clientSampling: true  # Always follow incoming trace context

Span fields

Add custom attributes to all spans, or remove default attributes:

config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: true
    fields:
      # Agentgateway implements OTel v1.37.0 gen-ai span attributes by default
      # See https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#inference
      add: {}

Access log export via OTLP

Export access logs as OpenTelemetry LogRecords alongside the default stdout output:

frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
  accessLog:
    otlp:
      host: localhost:4317
binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - mcp:
          targets:
          - name: everything
            stdio:
              cmd: npx
              args: ["@modelcontextprotocol/server-everything"]

Prometheus metrics

Metrics are enabled by default and exposed at /metrics on a dedicated stats address. Configure the address with config.statsAddr:

config:
  statsAddr: "0.0.0.0:15020"

If statsAddr is not set, agentgateway defaults to an internal address. Scrape the endpoint to verify:

curl localhost:15020/metrics -s | grep -v '#'

Example output:

tool_calls_total{server="everything",name="echo"} 1
tool_calls_total{server="everything",name="add"} 1
list_calls_total{resource_type="tool"} 3
agentgateway_requests_total{gateway="bind/3000",method="POST",status="200"} 9
agentgateway_requests_total{gateway="bind/3000",method="POST",status="202"} 4
agentgateway_requests_total{gateway="bind/3000",method="GET",status="200"} 1
agentgateway_requests_total{gateway="bind/3000",method="DELETE",status="202"} 2

Key metrics

Metric	Description
tool_calls_total	Total MCP tool calls, labelled by server and tool name
list_calls_total	Total MCP list operations, labelled by resource type
agentgateway_requests_total	Total HTTP requests, labelled by gateway, method, and status

Kubernetes scraping

The Helm chart configures Prometheus scraping annotations on pods by default:

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "9092"

Structured logging

Configure log level and format under config.logging:

config:
  logging:
    level: info
    format: json

Logging fields

Field	Description
level	Log level in RUST_LOG syntax. Defaults to info. Supports per-module levels, e.g. rmcp=warn,info
format	Log output format: json or text
fields.add	Additional fields to include in every log line
fields.remove	Fields to strip from log output
filter	Additional filter expression

Log level examples

config:
  logging:
    # Global info, quiet noisy modules
    level: "info"

    # More verbose — debug everything
    # level: "debug"

    # Per-module control
    # level: "rmcp=warn,hickory_server::server::server_future=off,info"

Running the telemetry example

1. Start a local Jaeger instance

   docker compose -f - up -d <<EOF
   services:
     jaeger:
       container_name: jaeger
       restart: unless-stopped
       image: jaegertracing/all-in-one:latest
       ports:
       - "127.0.0.1:16686:16686"
       - "127.0.0.1:4317:4317"
       environment:
       - COLLECTOR_OTLP_ENABLED=true
   EOF
   
2. Start agentgateway with tracing enabled

   cargo run -- -f examples/telemetry/config.yaml
   
3. Send requests via MCP Inspector

   npx @modelcontextprotocol/inspector
   

   Connect to http://localhost:3000 and invoke a few tools to generate trace data.

4. View traces in Jaeger

   Open http://localhost:16686/search and search for agentgateway spans.

5. Check metrics

   curl localhost:15020/metrics -s | grep -v '#'
   

OpenTelemetry Collector

For production deployments, route telemetry through the OpenTelemetry Collector. The example otel-collector-config.yaml fans out traces to Jaeger and logs to stdout:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch: {}

exporters:
  debug:
    verbosity: detailed
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

Grafana integration

A Grafana dashboard JSON file is included at manifests/grafana.json. Import it into your Grafana instance to visualize agentgateway metrics alongside traces from Jaeger or Grafana Tempo.

Agentgateway implements the OpenTelemetry Gen AI semantic conventions v1.37.0 for AI-specific span attributes, including model name, token counts, and provider information.