> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/agentgateway/agentgateway/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM context

> CEL reference for the llm context object — model, token counts, prompt, completion, and parameters available for AI backend policies.

# LLM context

The `llm` context object is available in CEL expressions when Agentgateway is proxying requests to an AI backend (`backend.type == "ai"`). It exposes information about the LLM request and response, including the model used, token counts, and the raw prompt and completion.

<Note>
  The `llm` object is only present when using an `ai` backend. Token count fields such as `llm.inputTokens` and `llm.outputTokens` are only populated after the response is received from the LLM provider.
</Note>

## Core fields

<ParamField path="llm.streaming" type="boolean" required>
  Whether the LLM response is being streamed.

  ```cel theme={null}
  # Only apply a policy for non-streaming requests
  !llm.streaming

  # Log differently based on streaming mode
  llm.streaming ? "stream" : "batch"
  ```
</ParamField>

<ParamField path="llm.requestModel" type="string" required>
  The model name requested by the client. This may differ from the model that actually served the response (see `llm.responseModel`).

  ```cel theme={null}
  llm.requestModel == "gpt-4o"
  llm.requestModel.startsWith("gpt-4")
  ```
</ParamField>

<ParamField path="llm.responseModel" type="string">
  The model that actually served the LLM response. May differ from `llm.requestModel` if the provider mapped the requested model to a different version.

  ```cel theme={null}
  llm.responseModel == "gpt-4o-2024-08-06"
  ```
</ParamField>

<ParamField path="llm.provider" type="string" required>
  The name of the LLM provider handling the request.

  ```cel theme={null}
  llm.provider == "openai"
  llm.provider == "anthropic"
  ```
</ParamField>

***

## Token counts

Token count fields are populated from the LLM provider's response and are available after the response is received (for example, in logging and post-response authorization policies).

<ParamField path="llm.inputTokens" type="integer">
  The number of tokens in the input prompt as reported by the provider.

  ```cel theme={null}
  llm.inputTokens > 10000
  ```
</ParamField>

<ParamField path="llm.cachedInputTokens" type="integer">
  The number of input tokens served from the provider's prompt cache. These represent cost savings.

  ```cel theme={null}
  llm.cachedInputTokens > 0
  ```
</ParamField>

<ParamField path="llm.cacheCreationInputTokens" type="integer">
  The number of tokens written to the provider's prompt cache. These represent additional cost for cache creation.

  <Note>
    `llm.cacheCreationInputTokens` is not present when using OpenAI. It is specific to providers that support explicit cache creation (such as Anthropic).
  </Note>

  ```cel theme={null}
  llm.cacheCreationInputTokens > 0
  ```
</ParamField>

<ParamField path="llm.outputTokens" type="integer">
  The number of tokens in the LLM response.

  ```cel theme={null}
  llm.outputTokens > 5000
  ```
</ParamField>

<ParamField path="llm.reasoningTokens" type="integer">
  The number of reasoning tokens in the output. Only populated for models that support extended reasoning (such as `o1` or Claude with thinking enabled).

  ```cel theme={null}
  llm.reasoningTokens > 0
  ```
</ParamField>

<ParamField path="llm.totalTokens" type="integer">
  The total number of tokens for the request (input + output).

  ```cel theme={null}
  llm.totalTokens > 20000
  ```
</ParamField>

<ParamField path="llm.countTokens" type="integer">
  The number of tokens counted when using the token counting endpoint. These are not counted as input tokens since the token counting endpoint does not consume tokens.

  ```cel theme={null}
  llm.countTokens > 50000
  ```
</ParamField>

***

## Prompt and completion

<ParamField path="llm.prompt" type="object[]">
  The prompt sent to the LLM as an array of chat messages. Each message has `role` and `content` fields.

  <Warning>
    Accessing `llm.prompt` has performance implications for large prompts, as the data must be retained in memory. Only reference this field when necessary.
  </Warning>

  <Expandable title="Message fields">
    <ParamField path="llm.prompt[].role" type="string">
      The role of the message sender. Typically `system`, `user`, or `assistant`.
    </ParamField>

    <ParamField path="llm.prompt[].content" type="string">
      The text content of the message.
    </ParamField>
  </Expandable>

  ```cel theme={null}
  # Check the role of the first message
  llm.prompt[0].role == "system"

  # Check if any message contains a keyword
  llm.prompt.exists(m, m.content.contains("confidential"))

  # Count user messages
  llm.prompt.filter(m, m.role == "user").size() > 10
  ```
</ParamField>

<ParamField path="llm.completion" type="string[]">
  The completion returned by the LLM as an array of strings.

  <Warning>
    Accessing `llm.completion` has performance implications for large responses, as the data must be retained in memory. Only reference this field when necessary.
  </Warning>

  ```cel theme={null}
  # Check if the completion contains sensitive content
  llm.completion.exists(c, c.contains("SSN"))
  ```
</ParamField>

***

## Parameters

The `llm.params` object contains the inference parameters from the LLM request.

<ParamField path="llm.params" type="object" required>
  The parameters for the LLM request.

  <Expandable title="Parameter fields">
    <ParamField path="llm.params.temperature" type="number">
      The sampling temperature. Higher values produce more random outputs.

      ```cel theme={null}
      llm.params.temperature > 1.0
      ```
    </ParamField>

    <ParamField path="llm.params.top_p" type="number">
      The nucleus sampling probability. Values closer to 1.0 include more of the probability distribution.

      ```cel theme={null}
      llm.params.top_p < 0.5
      ```
    </ParamField>

    <ParamField path="llm.params.frequency_penalty" type="number">
      Penalizes tokens based on how frequently they appear in the output so far.

      ```cel theme={null}
      has(llm.params.frequency_penalty)
      ```
    </ParamField>

    <ParamField path="llm.params.presence_penalty" type="number">
      Penalizes tokens based on whether they have appeared in the output at all.

      ```cel theme={null}
      has(llm.params.presence_penalty)
      ```
    </ParamField>

    <ParamField path="llm.params.seed" type="integer">
      The random seed for deterministic output.

      ```cel theme={null}
      has(llm.params.seed)
      ```
    </ParamField>

    <ParamField path="llm.params.max_tokens" type="integer">
      The maximum number of tokens to generate.

      ```cel theme={null}
      llm.params.max_tokens > 4096
      ```
    </ParamField>

    <ParamField path="llm.params.encoding_format" type="string">
      The encoding format for embeddings (for example, `float` or `base64`).

      ```cel theme={null}
      llm.params.encoding_format == "float"
      ```
    </ParamField>

    <ParamField path="llm.params.dimensions" type="integer">
      The number of dimensions for embedding models.

      ```cel theme={null}
      llm.params.dimensions == 1536
      ```
    </ParamField>
  </Expandable>
</ParamField>

***

## `llmRequest`

<ParamField path="llmRequest" type="object">
  The raw LLM request before any LLM policy processing. This is only available **during** LLM policy evaluation. Policies that run after the LLM policy — such as logging policies — will not have this field even for LLM requests.

  Use `llmRequest` when you need access to the unmodified request before transformations are applied.

  ```cel theme={null}
  # Access raw request fields during LLM policy processing
  has(llmRequest)
  ```
</ParamField>

***

## Examples

<AccordionGroup>
  <Accordion title="Rate limiting by total token count">
    Use token counts as the rate limiting key to enforce per-user token budgets. In a rate limiting policy, you can use CEL to select the dimension to limit on:

    ```cel theme={null}
    # Limit by user (from JWT) and total tokens
    jwt.sub
    ```

    Combined with threshold checks in authorization policies:

    ```cel theme={null}
    # Deny requests that would exceed a threshold (if pre-checked)
    llm.totalTokens > 100000
    ```
  </Accordion>

  <Accordion title="Log the model used and token breakdown">
    In a logging policy, define named fields using CEL expressions:

    ```cel theme={null}
    # provider: 'llm.provider'
    # requested_model: 'llm.requestModel'
    # actual_model: 'llm.responseModel'
    # input_tokens: 'llm.inputTokens'
    # output_tokens: 'llm.outputTokens'
    # total_tokens: 'llm.totalTokens'
    # cached_tokens: 'llm.cachedInputTokens'
    # streaming: 'llm.streaming'
    ```
  </Accordion>

  <Accordion title="Block high-temperature requests">
    Reject requests with a temperature above a threshold in an authorization policy:

    ```cel theme={null}
    !has(llm.params.temperature) || llm.params.temperature <= 1.5
    ```
  </Accordion>

  <Accordion title="Enforce max token limits">
    Reject requests that request more tokens than your policy allows:

    ```cel theme={null}
    !has(llm.params.max_tokens) || llm.params.max_tokens <= 4096
    ```
  </Accordion>

  <Accordion title="Restrict access to specific models">
    Only allow requests for approved models:

    ```cel theme={null}
    llm.requestModel in ["gpt-4o-mini", "gpt-3.5-turbo"]
    ```
  </Accordion>

  <Accordion title="Detect prompt content for policy enforcement">
    Check prompt messages for sensitive patterns. Use with prompt guard policies or custom authorization:

    ```cel theme={null}
    !llm.prompt.exists(m, m.content.matches("(?i)password|secret|api.key"))
    ```
  </Accordion>

  <Accordion title="Tracing: sample expensive requests">
    Sample only requests that exceed a token count threshold for detailed tracing:

    ```cel theme={null}
    llm.totalTokens > 5000 || random() < 0.01
    ```
  </Accordion>

  <Accordion title="Provider-specific logic">
    Apply different policies based on the provider:

    ```cel theme={null}
    # Anthropic-specific: check cache creation tokens
    llm.provider == "anthropic" && has(llm.cacheCreationInputTokens)

    # Require streaming for OpenAI to reduce cost
    llm.provider == "openai" && llm.streaming
    ```
  </Accordion>
</AccordionGroup>