> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/agentgateway/agentgateway/llms.txt
> Use this file to discover all available pages before exploring further.

# Telemetry

> Configure distributed tracing, Prometheus metrics, and structured logging for agentgateway.

Agentgateway emits telemetry through three channels:

* **Distributed traces** via OTLP to any OpenTelemetry-compatible backend (Jaeger, Grafana Tempo, etc.)
* **Prometheus metrics** scraped from a configurable stats address
* **Structured logs** written to stdout with configurable level and format

## Distributed tracing

Tracing is configured under `config.tracing` (static config file) or `frontendPolicies.tracing` (dynamic config). Send traces to any OTLP endpoint:

```yaml theme={null}
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: true
```

Or with the `frontendPolicies` form used in the telemetry example:

```yaml theme={null}
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - mcp:
          targets:
          - name: everything
            stdio:
              cmd: npx
              args: ["@modelcontextprotocol/server-everything"]
```

### Tracing fields

| Field            | Description                                                                                                      |
| ---------------- | ---------------------------------------------------------------------------------------------------------------- |
| `otlpEndpoint`   | Full OTLP endpoint URL, e.g. `http://localhost:4317`                                                             |
| `host`           | OTLP host in `host:port` format (used in `frontendPolicies`)                                                     |
| `otlpProtocol`   | OTLP transport protocol (`grpc` or `http/protobuf`)                                                              |
| `path`           | OTLP path. Defaults to `/v1/traces`                                                                              |
| `randomSampling` | CEL expression or boolean. Initiates a new trace when no incoming trace context is present. Defaults to `false`  |
| `clientSampling` | CEL expression or boolean. Whether to continue a trace from an incoming `traceparent` header. Defaults to `true` |
| `fields.add`     | Additional attributes to add to every span                                                                       |
| `fields.remove`  | Span attributes to strip before exporting                                                                        |

### Sampling

`randomSampling` controls whether agentgateway starts new traces for requests that arrive without a trace context. It accepts:

* `true` / `false` — sample all or none
* A float between `0.0` and `1.0` — percentage of requests to sample
* A CEL expression that evaluates to a float or boolean

`clientSampling` controls whether to propagate traces that arrive with an existing `traceparent` header. It defaults to `true` (always propagate).

```yaml theme={null}
config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: 0.1   # Sample 10% of new requests
    clientSampling: true  # Always follow incoming trace context
```

### Span fields

Add custom attributes to all spans, or remove default attributes:

```yaml theme={null}
config:
  tracing:
    otlpEndpoint: http://localhost:4317
    randomSampling: true
    fields:
      # Agentgateway implements OTel v1.37.0 gen-ai span attributes by default
      # See https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#inference
      add: {}
```

### Access log export via OTLP

Export access logs as OpenTelemetry LogRecords alongside the default stdout output:

```yaml theme={null}
frontendPolicies:
  tracing:
    host: localhost:4317
    randomSampling: true
  accessLog:
    otlp:
      host: localhost:4317
binds:
- port: 3000
  listeners:
  - routes:
    - backends:
      - mcp:
          targets:
          - name: everything
            stdio:
              cmd: npx
              args: ["@modelcontextprotocol/server-everything"]
```

## Prometheus metrics

Metrics are enabled by default and exposed at `/metrics` on a dedicated stats address. Configure the address with `config.statsAddr`:

```yaml theme={null}
config:
  statsAddr: "0.0.0.0:15020"
```

If `statsAddr` is not set, agentgateway defaults to an internal address. Scrape the endpoint to verify:

```bash theme={null}
curl localhost:15020/metrics -s | grep -v '#'
```

Example output:

```
tool_calls_total{server="everything",name="echo"} 1
tool_calls_total{server="everything",name="add"} 1
list_calls_total{resource_type="tool"} 3
agentgateway_requests_total{gateway="bind/3000",method="POST",status="200"} 9
agentgateway_requests_total{gateway="bind/3000",method="POST",status="202"} 4
agentgateway_requests_total{gateway="bind/3000",method="GET",status="200"} 1
agentgateway_requests_total{gateway="bind/3000",method="DELETE",status="202"} 2
```

### Key metrics

| Metric                        | Description                                                  |
| ----------------------------- | ------------------------------------------------------------ |
| `tool_calls_total`            | Total MCP tool calls, labelled by server and tool name       |
| `list_calls_total`            | Total MCP list operations, labelled by resource type         |
| `agentgateway_requests_total` | Total HTTP requests, labelled by gateway, method, and status |

### Kubernetes scraping

The Helm chart configures Prometheus scraping annotations on pods by default:

```yaml theme={null}
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "9092"
```

## Structured logging

Configure log level and format under `config.logging`:

```yaml theme={null}
config:
  logging:
    level: info
    format: json
```

### Logging fields

| Field           | Description                                                                                           |
| --------------- | ----------------------------------------------------------------------------------------------------- |
| `level`         | Log level in `RUST_LOG` syntax. Defaults to `info`. Supports per-module levels, e.g. `rmcp=warn,info` |
| `format`        | Log output format: `json` or `text`                                                                   |
| `fields.add`    | Additional fields to include in every log line                                                        |
| `fields.remove` | Fields to strip from log output                                                                       |
| `filter`        | Additional filter expression                                                                          |

### Log level examples

```yaml theme={null}
config:
  logging:
    # Global info, quiet noisy modules
    level: "info"

    # More verbose — debug everything
    # level: "debug"

    # Per-module control
    # level: "rmcp=warn,hickory_server::server::server_future=off,info"
```

## Running the telemetry example

<Steps>
  <Step title="Start a local Jaeger instance">
    ```bash theme={null}
    docker compose -f - up -d <<EOF
    services:
      jaeger:
        container_name: jaeger
        restart: unless-stopped
        image: jaegertracing/all-in-one:latest
        ports:
        - "127.0.0.1:16686:16686"
        - "127.0.0.1:4317:4317"
        environment:
        - COLLECTOR_OTLP_ENABLED=true
    EOF
    ```
  </Step>

  <Step title="Start agentgateway with tracing enabled">
    ```bash theme={null}
    cargo run -- -f examples/telemetry/config.yaml
    ```
  </Step>

  <Step title="Send requests via MCP Inspector">
    ```bash theme={null}
    npx @modelcontextprotocol/inspector
    ```

    Connect to `http://localhost:3000` and invoke a few tools to generate trace data.
  </Step>

  <Step title="View traces in Jaeger">
    Open [http://localhost:16686/search](http://localhost:16686/search) and search for `agentgateway` spans.
  </Step>

  <Step title="Check metrics">
    ```bash theme={null}
    curl localhost:15020/metrics -s | grep -v '#'
    ```
  </Step>
</Steps>

## OpenTelemetry Collector

For production deployments, route telemetry through the OpenTelemetry Collector. The example `otel-collector-config.yaml` fans out traces to Jaeger and logs to stdout:

```yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch: {}

exporters:
  debug:
    verbosity: detailed
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]
```

## Grafana integration

A Grafana dashboard JSON file is included at `manifests/grafana.json`. Import it into your Grafana instance to visualize agentgateway metrics alongside traces from Jaeger or Grafana Tempo.

<Note>
  Agentgateway implements the [OpenTelemetry Gen AI semantic conventions v1.37.0](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#inference) for AI-specific span attributes, including model name, token counts, and provider information.
</Note>