Skip to main content

Telemetry-verified tool calls

An eval normally scores the reply your agent returns. Telemetry-verified tool calls add a second check: they confirm that your agent actually invoked the tools an entry expected, using traces your agent exports to Spec27. An entry passes only when its answer scores correct and its expected tool calls appear in the telemetry for that run.

This is a paid feature. If your plan does not include it, the Telemetry card on the project overview shows an upgrade link, and testset entries do not offer expected tool calls.

Connect your agent's telemetry​

You connect an agent once per project. The project overview has a Telemetry card with everything you need.

  1. Open the project overview and find the Telemetry card.

  2. Select Generate exporter config. Spec27 mints a project API key and fills it into a ready-made exporter configuration.

  3. Copy the configuration into your agent. Two flavours are shown:

    Environment variables, for an OpenTelemetry SDK that reads them directly:

    OTEL_EXPORTER_OTLP_ENDPOINT=https://dashboard.spec27.ai/api/v1/otlp
    OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <your-key>

    OpenTelemetry Collector, if you route spans through a collector:

    exporters:
      otlphttp/spec27:
        endpoint: https://dashboard.spec27.ai/api/v1/otlp
        headers:
          Authorization: "Bearer <your-key>"

The key is shown once. Store it in your secret manager before you leave the page. If you lose it, generate a new one.

The Telemetry card shows no connection status until your first traces arrive. Once they do, it reads Receiving with the time telemetry was last seen. This status is the place to confirm your exporter is working.

Declare expected tool calls on an entry​

Expected tool calls live on a testset entry, alongside its input and expected output.

  1. Open the specification editor and add or edit an entry.
  2. In Expected tool calls, type a tool name and press Enter to add it as a chip. Remove a chip to drop the expectation.

An expectation passes when the tool is called at least once anywhere in the entry's conversation, across all turns. Expectations are conversation-scoped. There is no per-turn or ordering syntax.

Read the result​

When an entry has expected tool calls, its result shows an Expected tool calls checklist.

  • A pass shows a tick. Open it to see the observed call: the tool name, the arguments Spec27 received, and the time the call ran.
  • A fail shows a cross with one of two reasons. Not called means telemetry arrived for the run but the tool was not among it. No telemetry received means no traces arrived for the run, so the exporter is likely misconfigured or disconnected.
  • A settling state appears briefly while Spec27 waits for exported spans, which arrive shortly after a run completes.

If you see No telemetry received across results, check the connection status on the Telemetry card and confirm your exporter still points at the endpoint with a valid key.